[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:02 dist-test-slave-g350 krb5kdc[16232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:02 dist-test-slave-g350 krb5kdc[16232](info): set up 2 sockets
Aug 10 04:16:02 dist-test-slave-g350 krb5kdc[16232](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 04:16:03.597108 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:03 dist-test-slave-g350 krb5kdc[16232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:03.738543 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45105
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:03.827500 16246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:03.827840 16246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:03.827893 16246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:03.829303 16246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:03.829402 16246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:03.829432 16246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:03.829455 16246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:03.829490 16246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:03.831336 16246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:03.831689 16246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:03.832082 16246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:03.838032 16252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:03.838034 16254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:03.838095 16251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:03.838505 16246 server_base.cc:1034] running on GCE node
I20260810 04:16:03.838827 16246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:03.839181 16246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:03.840418 16246 hybrid_clock.cc:648] HybridClock initialized: now 1786335363840363 us; error 103 us; skew 500 ppm
Aug 10 04:16:03 dist-test-slave-g350 krb5kdc[16232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335363, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:03.843408 16246 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:03.843925 16246 webserver.cc:460] Webserver started at http://127.15.216.126:33149/ using document root <none> and password file <none>
I20260810 04:16:03.844156 16246 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:03.844202 16246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:03.844302 16246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:03.845541 16246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "c101af23d22a490d8fd92a0717905c74"
format_stamp: "Formatted at 2026-08-10 04:16:03 on dist-test-slave-g350"
server_key: "9744852dd903c4a633f60f5e2675a06a"
server_key_iv: "9036b79a2cbefb7ff715c4c5b214af66"
server_key_version: "encryptionkey@0"
I20260810 04:16:03.846184 16246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "c101af23d22a490d8fd92a0717905c74"
format_stamp: "Formatted at 2026-08-10 04:16:03 on dist-test-slave-g350"
server_key: "9744852dd903c4a633f60f5e2675a06a"
server_key_iv: "9036b79a2cbefb7ff715c4c5b214af66"
server_key_version: "encryptionkey@0"
I20260810 04:16:03.847697 16246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:03.848506 16261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:03.848717 16246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:03.848822 16246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "c101af23d22a490d8fd92a0717905c74"
format_stamp: "Formatted at 2026-08-10 04:16:03 on dist-test-slave-g350"
server_key: "9744852dd903c4a633f60f5e2675a06a"
server_key_iv: "9036b79a2cbefb7ff715c4c5b214af66"
server_key_version: "encryptionkey@0"
I20260810 04:16:03.848886 16246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:03.863147 16246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:03.865942 16246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:03.866071 16246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:03.870806 16246 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:45105
I20260810 04:16:03.870836 16315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:45105 every 8 connection(s)
I20260810 04:16:03.871240 16246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:03.872165 16316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:03.874933 16316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74: Bootstrap starting.
I20260810 04:16:03.875253 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 16246
I20260810 04:16:03.875339 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:03.875653 16316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:03.875989 16316 log.cc:824] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:03.876042 16225 external_mini_cluster.cc:1428] Setting key bd6eaf07f329ee8c19dc25740c5f8a40
I20260810 04:16:03.876854 16316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74: No bootstrap required, opened a new log
I20260810 04:16:03.878997 16316 raft_consensus.cc:348] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c101af23d22a490d8fd92a0717905c74" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45105 } }
I20260810 04:16:03.879184 16316 raft_consensus.cc:374] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:03.879237 16316 raft_consensus.cc:729] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c101af23d22a490d8fd92a0717905c74, State: Initialized, Role: FOLLOWER
I20260810 04:16:03.879395 16316 consensus_queue.cc:260] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c101af23d22a490d8fd92a0717905c74" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45105 } }
I20260810 04:16:03.879469 16316 raft_consensus.cc:388] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:03.879513 16316 raft_consensus.cc:482] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:03.879601 16316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:03.880827 16316 raft_consensus.cc:504] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c101af23d22a490d8fd92a0717905c74" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45105 } }
I20260810 04:16:03.880955 16316 leader_election.cc:302] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c101af23d22a490d8fd92a0717905c74; no voters:
I20260810 04:16:03.881157 16316 leader_election.cc:288] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:03.881220 16321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:03.881371 16321 raft_consensus.cc:686] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [term 1 LEADER]: Becoming Leader. State: Replica: c101af23d22a490d8fd92a0717905c74, State: Running, Role: LEADER
I20260810 04:16:03.881471 16316 sys_catalog.cc:564] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:03.881513 16321 consensus_queue.cc:237] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c101af23d22a490d8fd92a0717905c74" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45105 } }
I20260810 04:16:03.881970 16321 sys_catalog.cc:455] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c101af23d22a490d8fd92a0717905c74. Latest consensus state: current_term: 1 leader_uuid: "c101af23d22a490d8fd92a0717905c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c101af23d22a490d8fd92a0717905c74" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45105 } } }
I20260810 04:16:03.882273 16321 sys_catalog.cc:458] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:03.882169 16323 sys_catalog.cc:455] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c101af23d22a490d8fd92a0717905c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c101af23d22a490d8fd92a0717905c74" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45105 } } }
I20260810 04:16:03.882815 16323 sys_catalog.cc:458] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:03.882965 16331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:03.883500 16331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:03.885350 16331 catalog_manager.cc:1347] Generated new cluster ID: b9f3440ef20044b9a18ad41b5b4e501a
I20260810 04:16:03.885414 16331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:16:03 dist-test-slave-g350 krb5kdc[16232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:03.893750 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:03.877007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.15.216.126:45105)
0810 04:16:03.877225 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:03.877236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:03.877508 (+ 272us) server_negotiation.cc:407] Connection header received
0810 04:16:03.879294 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:03.879301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:03.879384 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:03.879552 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:03.882187 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:03.883788 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:03.885191 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:03.885795 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:03.885924 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:03.889583 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:03.889608 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:03.889608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:03.889622 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:03.891488 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:03.891943 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:03.891946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:03.891946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:03.892054 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:03.892341 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:03.892342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:03.892342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:03.892594 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:03.892770 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:03.893055 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 04:16:03.893410 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
I20260810 04:16:03.896217 16331 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:03.896827 16331 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:03.901741 16331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c101af23d22a490d8fd92a0717905c74: Generated new TSK 0
I20260810 04:16:03.901963 16331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:03.937099 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45105
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:04.033679 16345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:04.033950 16345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:04.034003 16345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:04.035620 16345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:04.035679 16345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:04.035714 16345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:04.037563 16345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:04.038153 16345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:04.038604 16345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:04.044281 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:04.044302 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:04.044322 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:04.044823 16345 server_base.cc:1034] running on GCE node
I20260810 04:16:04.045049 16345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:04.045311 16345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:04.046499 16345 hybrid_clock.cc:648] HybridClock initialized: now 1786335364046481 us; error 29 us; skew 500 ppm
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:04.049422 16345 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:04.050069 16345 webserver.cc:460] Webserver started at http://127.15.216.65:36587/ using document root <none> and password file <none>
I20260810 04:16:04.050338 16345 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:04.050413 16345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:04.050585 16345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:04.051793 16345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "ea9b27513198420a8ef09264e45f446a"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "e966d5047c06037f8ae95be7be70347c"
server_key_iv: "31230d278a62e7d80fca2e6eb00d6e34"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.052241 16345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "ea9b27513198420a8ef09264e45f446a"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "e966d5047c06037f8ae95be7be70347c"
server_key_iv: "31230d278a62e7d80fca2e6eb00d6e34"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.053869 16345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:04.054874 16361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.055080 16345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:04.055233 16345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "ea9b27513198420a8ef09264e45f446a"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "e966d5047c06037f8ae95be7be70347c"
server_key_iv: "31230d278a62e7d80fca2e6eb00d6e34"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.055330 16345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:04.074343 16345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:04.076619 16345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:04.076812 16345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:04.077075 16345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:04.077415 16345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:04.077477 16345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.077519 16345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:04.077550 16345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.084648 16345 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:35023
I20260810 04:16:04.084712 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:35023 every 8 connection(s)
I20260810 04:16:04.085053 16345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:04.085268 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 16345
I20260810 04:16:04.085372 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:04.085554 16225 external_mini_cluster.cc:1428] Setting key c34cff2e562c2955a0c371cd945a1e56
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
I20260810 04:16:04.096822 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.086022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:55855 (local address 127.15.216.126:45105)
0810 04:16:04.086228 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:04.086230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:04.086350 (+ 120us) server_negotiation.cc:407] Connection header received
0810 04:16:04.087155 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:04.087156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:04.087166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:04.087208 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:04.088709 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.089561 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.090623 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.090991 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.091081 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.093433 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:04.093435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:04.093435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:04.093444 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:04.095263 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.095715 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.095716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.095716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.095785 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.096061 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.096062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.096062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.096200 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:04.096283 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:04.096573 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 04:16:04.096719 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260810 04:16:04.097004 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.086048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45105 (local address 127.15.216.65:55855)
0810 04:16:04.086225 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:04.086270 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:04.087041 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:04.087281 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:04.087285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:04.088313 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:04.088577 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:04.088583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.089694 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.089695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:04.090520 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:04.090522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.091090 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.091091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:04.091209 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.091902 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:04.091911 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:04.093329 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:04.095361 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:04.095372 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:04.095374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:04.095618 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:04.095858 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:04.095859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:04.095860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:04.095986 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:04.096274 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:04.096283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:04.096428 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 04:16:04.096536 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:16:04.096725 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260810 04:16:04.097677 16481 heartbeater.cc:347] Connected to a master server at 127.15.216.126:45105
I20260810 04:16:04.097802 16481 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:04.098079 16481 heartbeater.cc:511] Master 127.15.216.126:45105 requested a full tablet report, sending...
I20260810 04:16:04.098901 16278 ts_manager.cc:194] Registered new tserver with Master: ea9b27513198420a8ef09264e45f446a (127.15.216.65:35023)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
I20260810 04:16:04.099538 16278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:55855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:04.118615 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45105
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:04.208618 16487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:04.208869 16487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:04.208899 16487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:04.210558 16487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:04.210632 16487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:04.210666 16487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:04.212339 16487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:04.212574 16487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:04.212776 16487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:04.218713 16494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:04.218676 16497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:04.218918 16495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:04.218773 16487 server_base.cc:1034] running on GCE node
I20260810 04:16:04.219210 16487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:04.219530 16487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:04.220687 16487 hybrid_clock.cc:648] HybridClock initialized: now 1786335364220665 us; error 41 us; skew 500 ppm
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:04.223505 16487 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:04.223974 16487 webserver.cc:460] Webserver started at http://127.15.216.66:40123/ using document root <none> and password file <none>
I20260810 04:16:04.224233 16487 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:04.224283 16487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:04.224435 16487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:04.225579 16487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "c44a0a6b668743d48e22096c736cf627"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "7ed1e7dffc1679e86eab66e5f11ce1f7"
server_key_iv: "d250c9bbd0c393d2dbe07c46ebe0dbc9"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.226049 16487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "c44a0a6b668743d48e22096c736cf627"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "7ed1e7dffc1679e86eab66e5f11ce1f7"
server_key_iv: "d250c9bbd0c393d2dbe07c46ebe0dbc9"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.227591 16487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:16:04.228492 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.228747 16487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:16:04.228856 16487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "c44a0a6b668743d48e22096c736cf627"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "7ed1e7dffc1679e86eab66e5f11ce1f7"
server_key_iv: "d250c9bbd0c393d2dbe07c46ebe0dbc9"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.228941 16487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:04.250479 16487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:04.253878 16487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:04.254058 16487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:04.254347 16487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:04.254714 16487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:04.254945 16487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.255033 16487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:04.255066 16487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.263645 16487 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:33585
I20260810 04:16:04.263700 16624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:33585 every 8 connection(s)
I20260810 04:16:04.264106 16487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:04.265342 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 16487
I20260810 04:16:04.265458 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:04.265681 16225 external_mini_cluster.cc:1428] Setting key 54fbcdf5d63c53c244814ccfdb36cbdd
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
I20260810 04:16:04.276856 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.265090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:57833 (local address 127.15.216.126:45105)
0810 04:16:04.265186 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:04.265189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:04.265456 (+ 267us) server_negotiation.cc:407] Connection header received
0810 04:16:04.266235 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:04.266236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:04.266248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:04.266316 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:04.267991 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.268693 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.270267 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.270621 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.270713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.273330 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:04.273332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:04.273332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:04.273342 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:04.275168 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.275676 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.275678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.275678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.275747 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.276102 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.276103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.276104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.276277 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:04.276365 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:04.276695 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 04:16:04.276782 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 04:16:04.277112 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.265020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45105 (local address 127.15.216.66:57833)
0810 04:16:04.265241 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:04.265359 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:04.266120 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:04.266431 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:04.266441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:04.267498 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:04.267858 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:04.267866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.268815 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.268816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:04.270160 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:04.270164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.270747 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.270748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:04.270900 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.271562 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:04.271575 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:04.273207 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:04.275275 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:04.275285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:04.275287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:04.275575 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:04.275872 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:04.275873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:04.275873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:04.276018 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:04.276367 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:04.276376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:04.276546 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 04:16:04.276641 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 04:16:04.276791 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":119,"threads_started":1}
I20260810 04:16:04.277691 16625 heartbeater.cc:347] Connected to a master server at 127.15.216.126:45105
I20260810 04:16:04.277812 16625 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:04.278072 16625 heartbeater.cc:511] Master 127.15.216.126:45105 requested a full tablet report, sending...
I20260810 04:16:04.278539 16278 ts_manager.cc:194] Registered new tserver with Master: c44a0a6b668743d48e22096c736cf627 (127.15.216.66:33585)
I20260810 04:16:04.279062 16278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:57833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:04.297565 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45105
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:04.387218 16630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:04.387449 16630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:04.387478 16630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:04.388711 16630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:04.388763 16630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:04.388793 16630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:04.390441 16630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:04.390722 16630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:04.390978 16630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:04.396772 16636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:04.396862 16630 server_base.cc:1034] running on GCE node
W20260810 04:16:04.396817 16638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:04.396772 16635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:04.397320 16630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:04.397691 16630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:04.398880 16630 hybrid_clock.cc:648] HybridClock initialized: now 1786335364398856 us; error 42 us; skew 500 ppm
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:04.402093 16630 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:04.402619 16630 webserver.cc:460] Webserver started at http://127.15.216.67:36321/ using document root <none> and password file <none>
I20260810 04:16:04.402868 16630 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:04.402912 16630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:04.403013 16630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:04.404120 16630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "e170cf71b1c34e4d9037e0ea8526b62c"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "5d28dbc53b14988d23f284d0cc2b4f19"
server_key_iv: "a5ec99e5ca232e5f843a070eaa45bea7"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.404440 16630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "e170cf71b1c34e4d9037e0ea8526b62c"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "5d28dbc53b14988d23f284d0cc2b4f19"
server_key_iv: "a5ec99e5ca232e5f843a070eaa45bea7"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.405907 16630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:16:04.406702 16645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.406904 16630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:04.407032 16630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "e170cf71b1c34e4d9037e0ea8526b62c"
format_stamp: "Formatted at 2026-08-10 04:16:04 on dist-test-slave-g350"
server_key: "5d28dbc53b14988d23f284d0cc2b4f19"
server_key_iv: "a5ec99e5ca232e5f843a070eaa45bea7"
server_key_version: "encryptionkey@0"
I20260810 04:16:04.407123 16630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:04.417023 16630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:04.419538 16630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:04.419718 16630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:04.419975 16630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:04.420313 16630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:04.420372 16630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.420414 16630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:04.420436 16630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:04.427253 16630 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:39069
I20260810 04:16:04.427338 16763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:39069 every 8 connection(s)
I20260810 04:16:04.427686 16630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:04.434135 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 16630
I20260810 04:16:04.434237 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:04.434408 16225 external_mini_cluster.cc:1428] Setting key 7702f1ef113eb2a709d8aefae6016533
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:04.438256 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.428672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:43949 (local address 127.15.216.126:45105)
0810 04:16:04.428775 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:04.428778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:04.428957 (+ 179us) server_negotiation.cc:407] Connection header received
0810 04:16:04.429511 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:04.429513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:04.429527 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:04.429629 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:04.430688 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.431198 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.432326 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.432533 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.432613 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.435481 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:04.435483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:04.435483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:04.435493 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:04.436931 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.437353 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.437353 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.437354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.437411 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.437684 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.437685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.437685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.437804 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:04.437860 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:04.438122 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 04:16:04.438188 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 04:16:04.438513 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.428568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45105 (local address 127.15.216.67:43949)
0810 04:16:04.428815 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:04.428867 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:04.429392 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:04.429690 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:04.429694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:04.430364 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:04.430567 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:04.430572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.431311 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.431312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:04.432223 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:04.432226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.432622 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.432623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:04.432749 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.433435 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:04.433444 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:04.435349 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:04.437007 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:04.437014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:04.437016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:04.437247 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:04.437473 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:04.437473 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:04.437474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:04.437578 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:04.437866 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:04.437872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:04.437996 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 04:16:04.438083 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:16:04.438217 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":129,"threads_started":1}
I20260810 04:16:04.439069 16764 heartbeater.cc:347] Connected to a master server at 127.15.216.126:45105
I20260810 04:16:04.439203 16764 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:04.439430 16764 heartbeater.cc:511] Master 127.15.216.126:45105 requested a full tablet report, sending...
I20260810 04:16:04.439858 16278 ts_manager.cc:194] Registered new tserver with Master: e170cf71b1c34e4d9037e0ea8526b62c (127.15.216.67:39069)
I20260810 04:16:04.440244 16278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:43949
I20260810 04:16:04.447069 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:04.474166 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.465117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.15.216.65:35023)
0810 04:16:04.465321 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:04.465327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:04.465340 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:16:04.465374 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:04.465376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:04.465391 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:04.465469 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:04.466203 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.466992 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.467816 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.468150 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.468213 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.470917 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:04.470928 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:04.470929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:04.470937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:04.472798 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.473175 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.473179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.473179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.473250 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.473506 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.473507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.473507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.473755 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:04.473879 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:04.473992 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:16:04.474070 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":109,"threads_started":1}
W20260810 04:16:04.474666 16390 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51768
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:04.500516 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:04.492381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.15.216.65:35023)
0810 04:16:04.492462 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:04.492465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:04.492599 (+ 134us) server_negotiation.cc:407] Connection header received
0810 04:16:04.492697 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:04.492698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:04.492712 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:04.492773 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:04.493402 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.493974 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.494777 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:04.494991 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:04.495072 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:04.497792 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:04.497794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:04.497794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:04.497802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:04.499217 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.499634 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.499635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.499635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.499703 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:04.499932 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:04.499933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:04.499933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:04.500045 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:04.500179 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:04.500308 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 04:16:04.500403 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:16:04.501164 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 16345
I20260810 04:16:04.509873 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 16487
I20260810 04:16:04.517230 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 16630
I20260810 04:16:04.523710 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 16246
2026-08-10T04:16:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2084 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16779](info): set up 2 sockets
Aug 10 04:16:04 dist-test-slave-g350 krb5kdc[16779](info): commencing operation
krb5kdc: starting...
W20260810 04:16:05.662288 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:05 dist-test-slave-g350 krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:05.797539 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40283
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:40283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:05.888862 16794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:05.889143 16794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:05.889195 16794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:05.890484 16794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:05.890563 16794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:05.890586 16794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:05.890616 16794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:05.890645 16794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:05.892200 16794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:40283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:05.892511 16794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:05.892774 16794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:05.898243 16802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:05.898237 16800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:05.898272 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:05.898468 16794 server_base.cc:1034] running on GCE node
I20260810 04:16:05.898793 16794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:05.899109 16794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:05.900339 16794 hybrid_clock.cc:648] HybridClock initialized: now 1786335365900305 us; error 73 us; skew 500 ppm
Aug 10 04:16:05 dist-test-slave-g350 krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335365, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:05.908346 16794 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:05.908900 16794 webserver.cc:460] Webserver started at http://127.15.216.126:33893/ using document root <none> and password file <none>
I20260810 04:16:05.909137 16794 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:05.909179 16794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:05.909274 16794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:05.910393 16794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "76e54216e0dd4376a31e225af557a5f8"
format_stamp: "Formatted at 2026-08-10 04:16:05 on dist-test-slave-g350"
server_key: "56cb2161dc9bc51c204a47391702ce82"
server_key_iv: "2ac68f72ab9c037fae5819cc2fb7829d"
server_key_version: "encryptionkey@0"
I20260810 04:16:05.910713 16794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "76e54216e0dd4376a31e225af557a5f8"
format_stamp: "Formatted at 2026-08-10 04:16:05 on dist-test-slave-g350"
server_key: "56cb2161dc9bc51c204a47391702ce82"
server_key_iv: "2ac68f72ab9c037fae5819cc2fb7829d"
server_key_version: "encryptionkey@0"
I20260810 04:16:05.912101 16794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:05.912909 16812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:05.913112 16794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:05.913203 16794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "76e54216e0dd4376a31e225af557a5f8"
format_stamp: "Formatted at 2026-08-10 04:16:05 on dist-test-slave-g350"
server_key: "56cb2161dc9bc51c204a47391702ce82"
server_key_iv: "2ac68f72ab9c037fae5819cc2fb7829d"
server_key_version: "encryptionkey@0"
I20260810 04:16:05.913352 16794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:05.923887 16794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:05.926295 16794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:05.926414 16794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:05.930678 16794 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:40283
I20260810 04:16:05.930733 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:40283 every 8 connection(s)
I20260810 04:16:05.931111 16794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:05.932206 16867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:05.933799 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 16794
I20260810 04:16:05.933871 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:05.934046 16225 external_mini_cluster.cc:1428] Setting key 7ce10b4bf6b1ef360a606d133d28e4a8
I20260810 04:16:05.934954 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8: Bootstrap starting.
I20260810 04:16:05.935621 16867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:05.935997 16867 log.cc:824] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:05.936920 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8: No bootstrap required, opened a new log
I20260810 04:16:05.939452 16867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e54216e0dd4376a31e225af557a5f8" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40283 } }
I20260810 04:16:05.939656 16867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:05.939694 16867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e54216e0dd4376a31e225af557a5f8, State: Initialized, Role: FOLLOWER
I20260810 04:16:05.939857 16867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e54216e0dd4376a31e225af557a5f8" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40283 } }
I20260810 04:16:05.939945 16867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:05.939975 16867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:05.940017 16867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:05.940835 16867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e54216e0dd4376a31e225af557a5f8" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40283 } }
I20260810 04:16:05.940937 16867 leader_election.cc:302] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e54216e0dd4376a31e225af557a5f8; no voters:
I20260810 04:16:05.941123 16867 leader_election.cc:288] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:05.941290 16873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:05.941406 16867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:05.941414 16873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [term 1 LEADER]: Becoming Leader. State: Replica: 76e54216e0dd4376a31e225af557a5f8, State: Running, Role: LEADER
I20260810 04:16:05.941519 16873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e54216e0dd4376a31e225af557a5f8" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40283 } }
Aug 10 04:16:05 dist-test-slave-g350 krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:05.943135 16873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76e54216e0dd4376a31e225af557a5f8. Latest consensus state: current_term: 1 leader_uuid: "76e54216e0dd4376a31e225af557a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e54216e0dd4376a31e225af557a5f8" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40283 } } }
I20260810 04:16:05.943209 16873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:05.943459 16886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:05.943986 16886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:05.947362 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:05.934645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44058 (local address 127.15.216.126:40283)
0810 04:16:05.934822 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:05.934829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:05.934856 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:16:05.935322 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:05.935330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:05.935393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:05.935508 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:05.936395 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:05.937542 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:05.938680 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:05.939276 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:05.939396 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:05.942667 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:05.942672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:05.942673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:05.942687 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:05.945351 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:05.945821 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:05.945825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:05.945825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:05.945945 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:05.946185 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:05.946186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:05.946186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:05.946511 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:05.946694 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:05.946963 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 04:16:05.947129 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260810 04:16:05.945669 16874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76e54216e0dd4376a31e225af557a5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e54216e0dd4376a31e225af557a5f8" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40283 } } }
I20260810 04:16:05.947983 16874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:05.950757 16886 catalog_manager.cc:1347] Generated new cluster ID: 6d1bcd547cdf49ca8b4f7d0061431f8b
I20260810 04:16:05.950809 16886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:05.964236 16886 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:05.964921 16886 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:05.969980 16886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76e54216e0dd4376a31e225af557a5f8: Generated new TSK 0
I20260810 04:16:05.970166 16886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:06.006695 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40283
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:06.093878 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:06.094095 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:06.094122 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:06.095427 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:06.095479 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:06.095508 16893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:06.097035 16893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:06.097249 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:06.097432 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:06.103121 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:06.103123 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:06.103235 16893 server_base.cc:1034] running on GCE node
W20260810 04:16:06.103127 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:06.103596 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:06.103880 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:06.105084 16893 hybrid_clock.cc:648] HybridClock initialized: now 1786335366105080 us; error 38 us; skew 500 ppm
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:06.108067 16893 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:06.108605 16893 webserver.cc:460] Webserver started at http://127.15.216.65:41361/ using document root <none> and password file <none>
I20260810 04:16:06.108879 16893 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:06.108956 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:06.109112 16893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:06.110342 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "7ea2755b551a4078adc12540b4f5ce34"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "02e62d73cf64643b266e0b1ce2484e2e"
server_key_iv: "0f533bae2e0db48c76884446fcf344a9"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.110747 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "7ea2755b551a4078adc12540b4f5ce34"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "02e62d73cf64643b266e0b1ce2484e2e"
server_key_iv: "0f533bae2e0db48c76884446fcf344a9"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.112144 16893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:06.113062 16910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.113323 16893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:06.113477 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "7ea2755b551a4078adc12540b4f5ce34"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "02e62d73cf64643b266e0b1ce2484e2e"
server_key_iv: "0f533bae2e0db48c76884446fcf344a9"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.113566 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:06.129164 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:06.131512 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:06.131691 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:06.131947 16893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:06.132282 16893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:06.132340 16893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.132383 16893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:06.132480 16893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.138744 16893 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:41037
I20260810 04:16:06.138790 17029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:41037 every 8 connection(s)
I20260810 04:16:06.139122 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:06.143146 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 16893
I20260810 04:16:06.143257 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:06.143466 16225 external_mini_cluster.cc:1428] Setting key 28cc0759e54e4e110c442136c8626404
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:06.150933 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.139896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:37961 (local address 127.15.216.126:40283)
0810 04:16:06.139962 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.139964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.140206 (+ 242us) server_negotiation.cc:407] Connection header received
0810 04:16:06.141230 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.141231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.141242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.141307 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:06.142145 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.142688 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.144220 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.144473 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.144533 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.147598 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:06.147600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:06.147600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:06.147608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:06.149339 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.149893 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.149894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.149894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.149956 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.150230 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.150231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.150231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.150359 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:06.150435 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.150734 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.150833 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:16:06.151122 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.139873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40283 (local address 127.15.216.65:37961)
0810 04:16:06.140068 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.140113 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.141103 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.141365 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.141368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.141770 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:06.141984 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.141988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.142839 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.142840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.144122 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.144126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.144577 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.144579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.144728 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.145327 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:06.145338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:06.147479 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:06.149441 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:06.149450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:06.149453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:06.149783 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:06.150026 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:06.150027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:06.150028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:06.150150 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:06.150440 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:06.150448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:06.150593 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.150699 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.150875 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":123,"threads_started":1}
I20260810 04:16:06.151649 17031 heartbeater.cc:347] Connected to a master server at 127.15.216.126:40283
I20260810 04:16:06.151750 17031 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:06.151973 17031 heartbeater.cc:511] Master 127.15.216.126:40283 requested a full tablet report, sending...
I20260810 04:16:06.152566 16829 ts_manager.cc:194] Registered new tserver with Master: 7ea2755b551a4078adc12540b4f5ce34 (127.15.216.65:41037)
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
I20260810 04:16:06.153194 16829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:37961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:06.174978 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40283
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:06.263348 17036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:06.263573 17036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:06.263602 17036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:06.264817 17036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:06.264870 17036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:06.264899 17036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:06.266968 17036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:06.267295 17036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:06.267794 17036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:06.273800 17041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:06.273938 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:06.274119 17036 server_base.cc:1034] running on GCE node
W20260810 04:16:06.273775 17042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:06.274557 17036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:06.274852 17036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:06.276055 17036 hybrid_clock.cc:648] HybridClock initialized: now 1786335366276033 us; error 64 us; skew 500 ppm
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:06.279343 17036 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:06.279888 17036 webserver.cc:460] Webserver started at http://127.15.216.66:42489/ using document root <none> and password file <none>
I20260810 04:16:06.280169 17036 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:06.280246 17036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:06.280407 17036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:06.281531 17036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "30889a5def5244f88d14fc07707a1ce1"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "874d1d032cf770ba37dff56fdbe9f005"
server_key_iv: "8d2e69180e1bc2e4c54be8da0f206ba4"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.281992 17036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "30889a5def5244f88d14fc07707a1ce1"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "874d1d032cf770ba37dff56fdbe9f005"
server_key_iv: "8d2e69180e1bc2e4c54be8da0f206ba4"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.283494 17036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:06.284408 17051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.284660 17036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:06.284808 17036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "30889a5def5244f88d14fc07707a1ce1"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "874d1d032cf770ba37dff56fdbe9f005"
server_key_iv: "8d2e69180e1bc2e4c54be8da0f206ba4"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.284899 17036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:06.295513 17036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:06.297947 17036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:06.298128 17036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:06.298398 17036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:06.298734 17036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:06.298792 17036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.298841 17036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:06.298869 17036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.305297 17036 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:33479
I20260810 04:16:06.305358 17172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:33479 every 8 connection(s)
I20260810 04:16:06.305802 17036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:06.311210 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 17036
I20260810 04:16:06.311329 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:06.311517 16225 external_mini_cluster.cc:1428] Setting key ad67372906dd5a901df5df45f1c3da2f
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:06.316594 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.306626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:46797 (local address 127.15.216.126:40283)
0810 04:16:06.306713 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.306716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.306913 (+ 197us) server_negotiation.cc:407] Connection header received
0810 04:16:06.307824 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.307825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.307839 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.307907 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:06.308515 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.309043 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.310176 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.310393 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.310453 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.312984 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:06.312985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:06.312986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:06.312993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:06.314830 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.315328 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.315329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.315330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.315404 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.315697 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.315698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.315699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.315854 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:06.315932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.316379 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.316470 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 04:16:06.316989 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.306566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40283 (local address 127.15.216.66:46797)
0810 04:16:06.306740 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.306780 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.307714 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.307963 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.307966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.308227 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:06.308425 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.308429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.309143 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.309144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.310096 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.310100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.310478 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.310479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.310580 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.311097 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:06.311106 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:06.312829 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:06.314963 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:06.314973 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:06.314975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:06.315233 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:06.315501 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:06.315502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:06.315502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:06.315620 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:06.315953 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:06.315962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:06.316166 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.316290 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.316644 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260810 04:16:06.317548 17173 heartbeater.cc:347] Connected to a master server at 127.15.216.126:40283
I20260810 04:16:06.317718 17173 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:06.317920 17173 heartbeater.cc:511] Master 127.15.216.126:40283 requested a full tablet report, sending...
I20260810 04:16:06.318398 16829 ts_manager.cc:194] Registered new tserver with Master: 30889a5def5244f88d14fc07707a1ce1 (127.15.216.66:33479)
I20260810 04:16:06.318951 16829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:46797
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:06.344388 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40283
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:06.430312 17178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:06.430521 17178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:06.430555 17178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:06.431665 17178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:06.431702 17178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:06.431766 17178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:06.433266 17178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:06.433462 17178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:06.433729 17178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:06.439265 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:06.439225 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:06.439323 17178 server_base.cc:1034] running on GCE node
W20260810 04:16:06.439297 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:06.439795 17178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:06.440071 17178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:06.441298 17178 hybrid_clock.cc:648] HybridClock initialized: now 1786335366441244 us; error 64 us; skew 500 ppm
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:06.444175 17178 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:06.444734 17178 webserver.cc:460] Webserver started at http://127.15.216.67:46253/ using document root <none> and password file <none>
I20260810 04:16:06.444936 17178 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:06.444979 17178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:06.445067 17178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:06.446266 17178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "cad9ffe65138436abe64d47803277d22"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "f437d7732d905061f98058f4ebfde144"
server_key_iv: "2880078e0ceb098b6db7b53c4b591cf0"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.446584 17178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "cad9ffe65138436abe64d47803277d22"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "f437d7732d905061f98058f4ebfde144"
server_key_iv: "2880078e0ceb098b6db7b53c4b591cf0"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.448045 17178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:06.448853 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.449033 17178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:06.449144 17178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "cad9ffe65138436abe64d47803277d22"
format_stamp: "Formatted at 2026-08-10 04:16:06 on dist-test-slave-g350"
server_key: "f437d7732d905061f98058f4ebfde144"
server_key_iv: "2880078e0ceb098b6db7b53c4b591cf0"
server_key_version: "encryptionkey@0"
I20260810 04:16:06.449224 17178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:06.469455 17178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:06.472093 17178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:06.472273 17178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:06.472558 17178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:06.472909 17178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:06.472968 17178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.473011 17178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:06.473032 17178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:06.479569 17178 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:45687
I20260810 04:16:06.479627 17319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:45687 every 8 connection(s)
I20260810 04:16:06.480012 17178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:06.480839 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 17178
I20260810 04:16:06.480932 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:06.481137 16225 external_mini_cluster.cc:1428] Setting key de1dfd5907ba7a4bd3aa72dec1d7cb6e
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:06.490353 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.480956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:46055 (local address 127.15.216.126:40283)
0810 04:16:06.481004 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.481006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.481251 (+ 245us) server_negotiation.cc:407] Connection header received
0810 04:16:06.482382 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.482383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.482396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.482456 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:06.483129 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.483645 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.484683 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.484888 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.484982 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.487419 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:06.487421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:06.487422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:06.487435 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:06.488775 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.489279 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.489281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.489281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.489339 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.489658 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.489659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.489659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.489786 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:06.489874 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.490160 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.490243 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:16:06.490561 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.480898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40283 (local address 127.15.216.67:46055)
0810 04:16:06.481096 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.481137 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.482260 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.482537 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.482541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.482826 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:06.483024 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.483028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.483761 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.483762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.484583 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.484585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.484997 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.484998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.485122 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.485706 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:06.485715 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:06.487289 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:06.488883 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:06.488891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:06.488893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:06.489188 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:06.489423 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:06.489424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:06.489425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:06.489541 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:06.489864 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:06.489872 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:06.490021 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.490139 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.490311 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
I20260810 04:16:06.491153 17320 heartbeater.cc:347] Connected to a master server at 127.15.216.126:40283
I20260810 04:16:06.491258 17320 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:06.491412 17320 heartbeater.cc:511] Master 127.15.216.126:40283 requested a full tablet report, sending...
I20260810 04:16:06.491798 16829 ts_manager.cc:194] Registered new tserver with Master: cad9ffe65138436abe64d47803277d22 (127.15.216.67:45687)
I20260810 04:16:06.492218 16829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:46055
I20260810 04:16:06.493084 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:16:06.501468 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.494889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.15.216.126:40283)
0810 04:16:06.494957 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.494959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.495048 (+ 89us) server_negotiation.cc:407] Connection header received
0810 04:16:06.495131 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.495132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.495142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.495194 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:06.495767 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.496291 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.497039 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.497264 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.497354 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.498654 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:06.498655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:06.498656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:06.498663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:06.500048 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.500435 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.500437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.500437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.500498 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.500714 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.500715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.500716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.500885 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:06.501004 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.501120 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.501210 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:16:06.504365 16828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:06.506132 16828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:06.513680 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.509165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.15.216.66:33479)
0810 04:16:06.509386 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.509392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.509620 (+ 228us) server_negotiation.cc:407] Connection header received
0810 04:16:06.509683 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.509686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.509709 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.509781 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:06.510375 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.511090 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.512701 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.513223 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.513321 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.513398 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.513511 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.513552 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260810 04:16:06.513696 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.509006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:33479 (local address 127.0.0.1:42278)
0810 04:16:06.509366 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.509410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.509496 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.509899 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.509902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.509904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:06.510102 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.510105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.511214 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.511215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.512601 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.512603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.513319 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.513320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.513403 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.513418 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.513495 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.513552 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":125,"thread_start_us":78,"threads_started":1}
I20260810 04:16:06.513901 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.509150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.15.216.65:41037)
0810 04:16:06.509479 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.509483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.509536 (+ 53us) server_negotiation.cc:407] Connection header received
0810 04:16:06.509749 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.509752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.509768 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.509849 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:06.510103 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.511292 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.512657 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.513362 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.513425 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.513474 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.513740 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.513783 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":102,"threads_started":1}
I20260810 04:16:06.513963 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.509006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:41037 (local address 127.0.0.1:51754)
0810 04:16:06.509454 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.509466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.509569 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.509900 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.509903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.509906 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:06.510008 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.510011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.511405 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.511406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.512563 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.512566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.513503 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.513504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.513564 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.513567 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.513642 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.513679 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":32,"thread_start_us":71,"threads_started":1}
I20260810 04:16:06.514107 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.509249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:45687 (local address 127.0.0.1:59146)
0810 04:16:06.509529 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.509538 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.509583 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.510196 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.510197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.510199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:06.510241 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.510243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.511459 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.511460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.512407 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.512413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.513448 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.513449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.513514 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.513519 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.513567 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.513642 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":90,"threads_started":1}
I20260810 04:16:06.514133 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.509367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.15.216.67:45687)
0810 04:16:06.509787 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.509791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.509798 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:06.509859 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.509861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.510024 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.510101 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:06.510320 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.511363 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.512515 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.513264 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.513337 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.513409 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.513740 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.513933 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":93,"threads_started":1}
I20260810 04:16:06.514726 17102 tablet_service.cc:1467] Processing CreateTablet for tablet 2e39653cbee7474997e8c3ab9920047f (DEFAULT_TABLE table=test-table [id=65a058bd350747c8ba8ff7d83292b256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:06.514729 16962 tablet_service.cc:1467] Processing CreateTablet for tablet 2e39653cbee7474997e8c3ab9920047f (DEFAULT_TABLE table=test-table [id=65a058bd350747c8ba8ff7d83292b256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:06.514829 17244 tablet_service.cc:1467] Processing CreateTablet for tablet 2e39653cbee7474997e8c3ab9920047f (DEFAULT_TABLE table=test-table [id=65a058bd350747c8ba8ff7d83292b256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:06.515103 17102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e39653cbee7474997e8c3ab9920047f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:06.515141 16962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e39653cbee7474997e8c3ab9920047f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:06.515141 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e39653cbee7474997e8c3ab9920047f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:06.518651 17338 tablet_bootstrap.cc:492] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1: Bootstrap starting.
I20260810 04:16:06.519135 17340 tablet_bootstrap.cc:492] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22: Bootstrap starting.
I20260810 04:16:06.519244 17339 tablet_bootstrap.cc:492] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34: Bootstrap starting.
I20260810 04:16:06.519681 17338 tablet_bootstrap.cc:654] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:06.519907 17339 tablet_bootstrap.cc:654] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:06.519906 17340 tablet_bootstrap.cc:654] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:06.520375 17338 log.cc:824] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:06.520538 17339 log.cc:824] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:06.520561 17340 log.cc:824] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:06.521332 17338 tablet_bootstrap.cc:492] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1: No bootstrap required, opened a new log
I20260810 04:16:06.521378 17340 tablet_bootstrap.cc:492] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22: No bootstrap required, opened a new log
I20260810 04:16:06.521430 17338 ts_tablet_manager.cc:1397] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:16:06.521462 17340 ts_tablet_manager.cc:1397] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:16:06.521646 17339 tablet_bootstrap.cc:492] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34: No bootstrap required, opened a new log
I20260810 04:16:06.521710 17339 ts_tablet_manager.cc:1397] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:16:06.523051 17340 raft_consensus.cc:348] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.523367 17340 raft_consensus.cc:374] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:06.523450 17340 raft_consensus.cc:729] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad9ffe65138436abe64d47803277d22, State: Initialized, Role: FOLLOWER
I20260810 04:16:06.523602 17340 consensus_queue.cc:260] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.523644 17338 raft_consensus.cc:348] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.523824 17338 raft_consensus.cc:374] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:06.523874 17338 raft_consensus.cc:729] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30889a5def5244f88d14fc07707a1ce1, State: Initialized, Role: FOLLOWER
I20260810 04:16:06.523756 17339 raft_consensus.cc:348] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.523936 17339 raft_consensus.cc:374] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:06.523970 17340 ts_tablet_manager.cc:1428] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:16:06.523980 17339 raft_consensus.cc:729] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ea2755b551a4078adc12540b4f5ce34, State: Initialized, Role: FOLLOWER
I20260810 04:16:06.524003 17338 consensus_queue.cc:260] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.524116 17320 heartbeater.cc:503] Master 127.15.216.126:40283 was elected leader, sending a full tablet report...
I20260810 04:16:06.524118 17339 consensus_queue.cc:260] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.524300 17338 ts_tablet_manager.cc:1428] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:06.524333 17173 heartbeater.cc:503] Master 127.15.216.126:40283 was elected leader, sending a full tablet report...
I20260810 04:16:06.524385 17339 ts_tablet_manager.cc:1428] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:06.524497 17031 heartbeater.cc:503] Master 127.15.216.126:40283 was elected leader, sending a full tablet report...
I20260810 04:16:06.525923 17344 raft_consensus.cc:482] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:06.526036 17344 raft_consensus.cc:504] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.526408 17344 leader_election.cc:288] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ea2755b551a4078adc12540b4f5ce34 (127.15.216.65:41037), 30889a5def5244f88d14fc07707a1ce1 (127.15.216.66:33479)
I20260810 04:16:06.529542 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.526599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:41037 (local address 127.15.216.67:49455)
0810 04:16:06.526758 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.526780 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.526842 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.526974 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.526975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.526976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:06.527050 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.527052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.527686 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.527687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.528675 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.528678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.529255 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.529256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.529328 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.529332 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.529409 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.529453 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260810 04:16:06.529528 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.526654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:49455 (local address 127.15.216.65:41037)
0810 04:16:06.526746 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.526748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.526855 (+ 107us) server_negotiation.cc:407] Connection header received
0810 04:16:06.526900 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.526901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.526910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.526958 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:06.527136 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.527597 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.528768 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.529165 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.529256 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.529273 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.529418 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.529456 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 04:16:06.529654 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.526694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:53437 (local address 127.15.216.66:33479)
0810 04:16:06.526741 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.526743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.526751 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:06.526776 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.526777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.526786 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.526878 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:06.527050 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.527546 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.528764 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.529180 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.529284 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.529295 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.529511 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.529554 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:16:06.529662 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.526570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:33479 (local address 127.15.216.67:53437)
0810 04:16:06.526614 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:06.526626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:06.526675 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:06.526885 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:06.526886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:06.526888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:06.526963 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.526965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.527646 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.527647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.528654 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:06.528657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.529309 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.529310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:06.529411 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.529414 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:06.529501 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:16:06.529548 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260810 04:16:06.530346 16983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e39653cbee7474997e8c3ab9920047f" candidate_uuid: "cad9ffe65138436abe64d47803277d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ea2755b551a4078adc12540b4f5ce34" is_pre_election: true
I20260810 04:16:06.530520 16983 raft_consensus.cc:2454] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad9ffe65138436abe64d47803277d22 in term 0.
I20260810 04:16:06.530809 17195 leader_election.cc:302] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ea2755b551a4078adc12540b4f5ce34, cad9ffe65138436abe64d47803277d22; no voters:
I20260810 04:16:06.530905 17123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e39653cbee7474997e8c3ab9920047f" candidate_uuid: "cad9ffe65138436abe64d47803277d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30889a5def5244f88d14fc07707a1ce1" is_pre_election: true
I20260810 04:16:06.530987 17344 raft_consensus.cc:2781] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:06.531042 17344 raft_consensus.cc:482] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:06.531042 17123 raft_consensus.cc:2454] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cad9ffe65138436abe64d47803277d22 in term 0.
I20260810 04:16:06.531076 17344 raft_consensus.cc:3037] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:06.531873 17344 raft_consensus.cc:504] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.532033 17344 leader_election.cc:288] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [CANDIDATE]: Term 1 election: Requested vote from peers 7ea2755b551a4078adc12540b4f5ce34 (127.15.216.65:41037), 30889a5def5244f88d14fc07707a1ce1 (127.15.216.66:33479)
I20260810 04:16:06.532251 17123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e39653cbee7474997e8c3ab9920047f" candidate_uuid: "cad9ffe65138436abe64d47803277d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30889a5def5244f88d14fc07707a1ce1"
I20260810 04:16:06.532269 16983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e39653cbee7474997e8c3ab9920047f" candidate_uuid: "cad9ffe65138436abe64d47803277d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ea2755b551a4078adc12540b4f5ce34"
I20260810 04:16:06.532353 17123 raft_consensus.cc:3037] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:06.532392 16983 raft_consensus.cc:3037] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:06.533362 16983 raft_consensus.cc:2454] T 2e39653cbee7474997e8c3ab9920047f P 7ea2755b551a4078adc12540b4f5ce34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad9ffe65138436abe64d47803277d22 in term 1.
I20260810 04:16:06.533357 17123 raft_consensus.cc:2454] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cad9ffe65138436abe64d47803277d22 in term 1.
I20260810 04:16:06.533577 17195 leader_election.cc:302] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ea2755b551a4078adc12540b4f5ce34, cad9ffe65138436abe64d47803277d22; no voters:
I20260810 04:16:06.533737 17344 raft_consensus.cc:2781] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:06.533897 17344 raft_consensus.cc:686] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [term 1 LEADER]: Becoming Leader. State: Replica: cad9ffe65138436abe64d47803277d22, State: Running, Role: LEADER
I20260810 04:16:06.533975 17344 consensus_queue.cc:237] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } }
I20260810 04:16:06.534878 16827 catalog_manager.cc:5495] T 2e39653cbee7474997e8c3ab9920047f P cad9ffe65138436abe64d47803277d22 reported cstate change: term changed from 0 to 1, leader changed from <none> to cad9ffe65138436abe64d47803277d22 (127.15.216.67). New cstate: current_term: 1 leader_uuid: "cad9ffe65138436abe64d47803277d22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2755b551a4078adc12540b4f5ce34" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 41037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30889a5def5244f88d14fc07707a1ce1" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cad9ffe65138436abe64d47803277d22" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 45687 } health_report { overall_health: HEALTHY } } }
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:06.547320 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.538657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.15.216.65:41037)
0810 04:16:06.538740 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.538742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.538826 (+ 84us) server_negotiation.cc:407] Connection header received
0810 04:16:06.538950 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.538951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.538964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.539040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:06.539619 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.540118 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.540934 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.541124 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.541185 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.544022 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:06.544035 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:06.544036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:06.544043 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:06.545849 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.546359 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.546363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.546363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.546425 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.546643 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.546644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.546644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.546809 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:06.546930 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.547109 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.547192 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 04:16:06.556458 17174 tablet.cc:2369] T 2e39653cbee7474997e8c3ab9920047f P 30889a5def5244f88d14fc07707a1ce1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:06.574594 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.566666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.15.216.65:41037)
0810 04:16:06.566763 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.566765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.566778 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:16:06.566830 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.566832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.566842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.566900 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:06.567536 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.568030 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.568832 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.569035 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.569096 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.571630 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:06.571631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:06.571631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:06.571639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:06.573212 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.573574 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.573575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.573575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.573698 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.573939 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.573939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.573940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.574046 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:06.574125 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.574315 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.574431 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[16779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:06.600265 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:06.592468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.15.216.65:41037)
0810 04:16:06.592537 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:06.592539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:06.592551 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:16:06.592606 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:06.592611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:06.592621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:06.592689 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:06.593307 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.593987 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.594788 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:06.595024 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:06.595106 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:06.597553 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:06.597555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:06.597555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:06.597562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:06.599048 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.599437 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.599438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.599438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.599497 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:06.599730 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:06.599731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:06.599731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:06.599870 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:06.599969 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:06.600119 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 04:16:06.600195 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 04:16:06.600960 16940 tablet_service.cc:2908] Rejecting scan request for tablet 2e39653cbee7474997e8c3ab9920047f: Uninitialized: clean time has not yet been initialized
I20260810 04:16:06.602500 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 16893
I20260810 04:16:06.610009 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 17036
I20260810 04:16:06.616307 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 17178
I20260810 04:16:06.622257 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 16794
2026-08-10T04:16:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2097 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[17362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[17362](info): set up 2 sockets
Aug 10 04:16:06 dist-test-slave-g350 krb5kdc[17362](info): commencing operation
krb5kdc: starting...
W20260810 04:16:07.727073 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:07 dist-test-slave-g350 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:07.865123 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:38241
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:07.951464 17377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:07.951704 17377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:07.951742 17377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:07.952915 17377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:07.952986 17377 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:16:07.953011 17377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:07.953043 17377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:07.953070 17377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:07.953095 17377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:07.954682 17377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:38241
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:38241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:07.954974 17377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:07.955209 17377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:07.960496 17382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:07.960482 17383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:07.960825 17377 server_base.cc:1034] running on GCE node
W20260810 04:16:07.960482 17385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:07.961170 17377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:07.961498 17377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:07.962672 17377 hybrid_clock.cc:648] HybridClock initialized: now 1786335367962647 us; error 39 us; skew 500 ppm
Aug 10 04:16:07 dist-test-slave-g350 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335367, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:07.965214 17377 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:07.965732 17377 webserver.cc:460] Webserver started at http://127.15.216.126:33295/ using document root <none> and password file <none>
I20260810 04:16:07.965988 17377 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:07.966060 17377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:07.966205 17377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:07.967274 17377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "cfac258c299f422baaf14d5eb4001455"
format_stamp: "Formatted at 2026-08-10 04:16:07 on dist-test-slave-g350"
server_key: "bc1f16669d87cbe946093892f3abd3cf"
server_key_iv: "64bd5f46a1909fc0bf6c42d96cb110b9"
server_key_version: "encryptionkey@0"
I20260810 04:16:07.967679 17377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "cfac258c299f422baaf14d5eb4001455"
format_stamp: "Formatted at 2026-08-10 04:16:07 on dist-test-slave-g350"
server_key: "bc1f16669d87cbe946093892f3abd3cf"
server_key_iv: "64bd5f46a1909fc0bf6c42d96cb110b9"
server_key_version: "encryptionkey@0"
I20260810 04:16:07.969008 17377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:07.969802 17393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:07.969982 17377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:07.970091 17377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "cfac258c299f422baaf14d5eb4001455"
format_stamp: "Formatted at 2026-08-10 04:16:07 on dist-test-slave-g350"
server_key: "bc1f16669d87cbe946093892f3abd3cf"
server_key_iv: "64bd5f46a1909fc0bf6c42d96cb110b9"
server_key_version: "encryptionkey@0"
I20260810 04:16:07.970166 17377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:07.978111 17377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:07.980378 17377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:07.980549 17377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:07.985647 17377 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:38241
I20260810 04:16:07.985721 17465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:38241 every 8 connection(s)
I20260810 04:16:07.986060 17377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:07.986743 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:07.988946 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455: Bootstrap starting.
I20260810 04:16:07.989485 17466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:07.989815 17466 log.cc:824] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:07.990531 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455: No bootstrap required, opened a new log
I20260810 04:16:07.991122 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 17377
I20260810 04:16:07.991225 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:07.991390 16225 external_mini_cluster.cc:1428] Setting key 96353c4cb7ade1c36c2312b8d981f9e5
I20260810 04:16:07.993160 17466 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfac258c299f422baaf14d5eb4001455" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38241 } }
I20260810 04:16:07.993338 17466 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:07.993427 17466 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfac258c299f422baaf14d5eb4001455, State: Initialized, Role: FOLLOWER
I20260810 04:16:07.993616 17466 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfac258c299f422baaf14d5eb4001455" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38241 } }
I20260810 04:16:07.993716 17466 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:07.993750 17466 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:07.993810 17466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:07.994472 17466 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfac258c299f422baaf14d5eb4001455" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38241 } }
I20260810 04:16:07.994562 17466 leader_election.cc:302] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfac258c299f422baaf14d5eb4001455; no voters:
I20260810 04:16:07.994755 17466 leader_election.cc:288] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:07.994798 17471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:07.994941 17471 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [term 1 LEADER]: Becoming Leader. State: Replica: cfac258c299f422baaf14d5eb4001455, State: Running, Role: LEADER
I20260810 04:16:07.995021 17466 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:07.995085 17471 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfac258c299f422baaf14d5eb4001455" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38241 } }
I20260810 04:16:07.995479 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfac258c299f422baaf14d5eb4001455. Latest consensus state: current_term: 1 leader_uuid: "cfac258c299f422baaf14d5eb4001455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfac258c299f422baaf14d5eb4001455" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38241 } } }
I20260810 04:16:07.995738 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:07.995662 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfac258c299f422baaf14d5eb4001455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfac258c299f422baaf14d5eb4001455" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38241 } } }
I20260810 04:16:07.996160 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:07.996341 17480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:07.996932 17480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:07.998629 17480 catalog_manager.cc:1347] Generated new cluster ID: c68a784d30264bde85f64433ebfd6ccc
I20260810 04:16:07.998682 17480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:16:07 dist-test-slave-g350 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:08.004488 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:07.991905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.15.216.126:38241)
0810 04:16:07.992095 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:07.992102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:07.992127 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:16:07.993139 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:07.993146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:07.993193 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:07.993306 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:07.994181 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:07.995404 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:07.996602 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:07.997204 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:07.997315 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.000179 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.000183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.000183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.000194 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.002157 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.002586 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.002588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.002589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.003247 (+ 658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.003503 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.003503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.003504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.003791 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.003964 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.004052 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.004290 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260810 04:16:08.021133 17480 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:08.021788 17480 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:08.025967 17480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455: Generated new TSK 0
I20260810 04:16:08.026176 17480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:08.065797 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38241
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:08.150245 17492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:08.150465 17492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:08.150489 17492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:08.151652 17492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:08.151696 17492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:08.151717 17492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:08.151736 17492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:08.153178 17492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:08.153427 17492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:08.153707 17492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:08.159241 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:08.159389 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:08.159335 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:08.159446 17492 server_base.cc:1034] running on GCE node
I20260810 04:16:08.159742 17492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:08.160033 17492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:08.161270 17492 hybrid_clock.cc:648] HybridClock initialized: now 1786335368161258 us; error 47 us; skew 500 ppm
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:08.164059 17492 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:08.164566 17492 webserver.cc:460] Webserver started at http://127.15.216.65:34705/ using document root <none> and password file <none>
I20260810 04:16:08.164836 17492 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:08.164913 17492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:08.165066 17492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:08.166308 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "0263215fa9dd4626b4179d630fd270ec"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "d0bbea8e6fa4c691754734a0be4a0ca9"
server_key_iv: "d8dc4e2ccf8c33218e5567a3925abce2"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.166674 17492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "0263215fa9dd4626b4179d630fd270ec"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "d0bbea8e6fa4c691754734a0be4a0ca9"
server_key_iv: "d8dc4e2ccf8c33218e5567a3925abce2"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.168036 17492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:08.168843 17507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.169023 17492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:08.169134 17492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "0263215fa9dd4626b4179d630fd270ec"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "d0bbea8e6fa4c691754734a0be4a0ca9"
server_key_iv: "d8dc4e2ccf8c33218e5567a3925abce2"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.169210 17492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:08.178973 17492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:08.180970 17492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:08.181135 17492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:08.181769 17492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:08.181811 17492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.181835 17492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:08.181875 17492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.188778 17492 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:37429
I20260810 04:16:08.188861 17630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:37429 every 8 connection(s)
I20260810 04:16:08.189152 17492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:08.191576 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 17492
I20260810 04:16:08.191701 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:08.191927 16225 external_mini_cluster.cc:1428] Setting key fa91c0a4458eecbb5f6d1e8a94602683
I20260810 04:16:08.193220 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.182424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:57427 (local address 127.15.216.126:38241)
0810 04:16:08.182482 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.182485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.182713 (+ 228us) server_negotiation.cc:407] Connection header received
0810 04:16:08.183278 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.183280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.183298 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.183357 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.184255 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.185150 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.186834 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.187190 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.187290 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.190053 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.190055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.190055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.190066 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.191498 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.192068 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.192070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.192070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.192149 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.192454 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.192455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.192455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.192599 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.192678 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.193062 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.193153 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:16:08.193483 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.182407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38241 (local address 127.15.216.65:57427)
0810 04:16:08.182589 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.182626 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.183178 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.183433 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.183438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.183816 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:08.184115 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.184121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.185271 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.185273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.186731 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.186736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.187300 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.187301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.187466 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.188118 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:08.188128 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:08.189934 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:08.191629 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.191638 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.191640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.191965 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.192229 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.192230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.192230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.192380 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.192681 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:08.192689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:08.192860 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.193008 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.193192 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260810 04:16:08.193990 17631 heartbeater.cc:347] Connected to a master server at 127.15.216.126:38241
I20260810 04:16:08.194108 17631 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:08.194299 17631 heartbeater.cc:511] Master 127.15.216.126:38241 requested a full tablet report, sending...
I20260810 04:16:08.195257 17412 ts_manager.cc:194] Registered new tserver with Master: 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429)
I20260810 04:16:08.196051 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:57427
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
I20260810 04:16:08.202669 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.196447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:59081 (local address 127.15.216.126:38241)
0810 04:16:08.196489 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.196491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.196615 (+ 124us) server_negotiation.cc:407] Connection header received
0810 04:16:08.196710 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.196711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.196720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.196750 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.197224 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.197791 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.198495 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.198707 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.198787 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.199834 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.199836 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.199836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.199844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.201312 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.201803 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.201805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.201805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.201871 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.202087 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.202089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.202089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.202211 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.202291 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.202494 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.202575 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:16:08.202669 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.196385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38241 (local address 127.15.216.65:59081)
0810 04:16:08.196551 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.196562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.196640 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.196867 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.196868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.197070 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:08.197130 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.197132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.197889 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.197890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.198393 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.198394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.198788 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.198789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.198866 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.199399 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:08.199404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:08.199703 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:08.201443 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.201444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.201445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.201718 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.201950 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.201951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.201951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.202021 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.202302 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:08.202304 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:08.202353 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.202436 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.202562 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:08.221278 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38241
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:08.307482 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:08.307714 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:08.307754 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:08.308913 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:08.308983 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:08.309018 17640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:08.309052 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:08.310498 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:08.310766 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:08.311009 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:08.316425 17649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:08.316402 17647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:08.316402 17646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:08.316624 17640 server_base.cc:1034] running on GCE node
I20260810 04:16:08.316949 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:08.317155 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:08.318419 17640 hybrid_clock.cc:648] HybridClock initialized: now 1786335368318409 us; error 32 us; skew 500 ppm
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:08.321012 17640 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:08.321494 17640 webserver.cc:460] Webserver started at http://127.15.216.66:43269/ using document root <none> and password file <none>
I20260810 04:16:08.321780 17640 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:08.321856 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:08.322010 17640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:08.323065 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "c9899dbe6205420c8eb0648991b33e27"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "826e4e4ba4609019f9755e4e56da4a41"
server_key_iv: "7ce241b80c01b025e6a826f8473394fa"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.323477 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "c9899dbe6205420c8eb0648991b33e27"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "826e4e4ba4609019f9755e4e56da4a41"
server_key_iv: "7ce241b80c01b025e6a826f8473394fa"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.324800 17640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:08.325560 17656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.325763 17640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:08.325868 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "c9899dbe6205420c8eb0648991b33e27"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "826e4e4ba4609019f9755e4e56da4a41"
server_key_iv: "7ce241b80c01b025e6a826f8473394fa"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.325953 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:08.337421 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:08.339560 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:08.339731 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:08.340267 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:08.340337 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.340373 17640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:08.340401 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.347545 17640 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:35707
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:08.347645 17787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:35707 every 8 connection(s)
I20260810 04:16:08.347972 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:08.350870 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.340764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:58379 (local address 127.15.216.126:38241)
0810 04:16:08.340807 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.340809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.341066 (+ 257us) server_negotiation.cc:407] Connection header received
0810 04:16:08.341665 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.341666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.341680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.341733 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.342610 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.343409 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.344940 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.345248 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.345315 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.348085 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.348086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.348087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.348096 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.349428 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.349879 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.349881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.349881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.349945 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.350212 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.350213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.350213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.350330 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.350411 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.350692 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.350779 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:16:08.351048 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.340774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38241 (local address 127.15.216.66:58379)
0810 04:16:08.340946 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.340981 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.341524 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.341817 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.341820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.342208 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:08.342514 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.342521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.343495 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.343496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.344862 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.344866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.345344 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.345345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.345496 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.346159 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:08.346169 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:08.347983 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:08.349516 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.349525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.349527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.349798 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.350016 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.350017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.350017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.350136 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.350407 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:08.350415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:08.350553 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.350652 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.350810 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260810 04:16:08.351557 17788 heartbeater.cc:347] Connected to a master server at 127.15.216.126:38241
I20260810 04:16:08.351641 17788 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:08.351824 17788 heartbeater.cc:511] Master 127.15.216.126:38241 requested a full tablet report, sending...
I20260810 04:16:08.352203 17412 ts_manager.cc:194] Registered new tserver with Master: c9899dbe6205420c8eb0648991b33e27 (127.15.216.66:35707)
I20260810 04:16:08.352615 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:58379
I20260810 04:16:08.356995 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 17640
I20260810 04:16:08.357066 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:08.357256 16225 external_mini_cluster.cc:1428] Setting key a84464618e4aba33d35f74647cf0606b
I20260810 04:16:08.358831 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.352688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:36641 (local address 127.15.216.126:38241)
0810 04:16:08.352758 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.352760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.352880 (+ 120us) server_negotiation.cc:407] Connection header received
0810 04:16:08.352951 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.352951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.352978 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.353032 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.353320 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.353923 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.354765 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.354939 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.354988 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.355938 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.355939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.355939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.355945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.357635 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.358020 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.358021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.358022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.358079 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.358288 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.358289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.358289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.358490 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.358561 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.358715 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.358772 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:16:08.358979 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.352625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38241 (local address 127.15.216.66:36641)
0810 04:16:08.352819 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.352830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.352894 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.353039 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.353040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.353191 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:08.353244 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.353246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.354056 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.354057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.354677 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.354680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.355032 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.355033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.355098 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.355563 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:08.355567 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:08.355840 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:08.357709 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.357710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.357711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.357927 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.358147 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.358148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.358148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.358224 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.358561 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:08.358562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:08.358615 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.358697 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.358822 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:08.385869 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38241
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:41813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:08.468364 17799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:08.468580 17799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:08.468604 17799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:08.469901 17799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:08.469980 17799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:08.470018 17799 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:08.470055 17799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:08.471578 17799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:38241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:08.471848 17799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:08.472091 17799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:08.477349 17804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:08.477349 17805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:08.477427 17807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:08.477488 17799 server_base.cc:1034] running on GCE node
I20260810 04:16:08.477823 17799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:08.478085 17799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:08.479266 17799 hybrid_clock.cc:648] HybridClock initialized: now 1786335368479241 us; error 39 us; skew 500 ppm
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:08.482173 17799 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:08.482590 17799 webserver.cc:460] Webserver started at http://127.15.216.67:36229/ using document root <none> and password file <none>
I20260810 04:16:08.482780 17799 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:08.482822 17799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:08.482911 17799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:08.483949 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "7bea558de0ef4167bb991bb324e7a6fa"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "bd126fa75ca049343f5340ad3eb36ebc"
server_key_iv: "ae11b6ea0268b6d16ece9a745a26b5ea"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.484261 17799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "7bea558de0ef4167bb991bb324e7a6fa"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "bd126fa75ca049343f5340ad3eb36ebc"
server_key_iv: "ae11b6ea0268b6d16ece9a745a26b5ea"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.485526 17799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:16:08.486373 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.486539 17799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:08.486632 17799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "7bea558de0ef4167bb991bb324e7a6fa"
format_stamp: "Formatted at 2026-08-10 04:16:08 on dist-test-slave-g350"
server_key: "bd126fa75ca049343f5340ad3eb36ebc"
server_key_iv: "ae11b6ea0268b6d16ece9a745a26b5ea"
server_key_version: "encryptionkey@0"
I20260810 04:16:08.486690 17799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:08.498310 17799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:08.500379 17799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:08.500500 17799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:08.500945 17799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:08.500981 17799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.501003 17799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:08.501061 17799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.508297 17799 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:33805
I20260810 04:16:08.508410 17940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:33805 every 8 connection(s)
I20260810 04:16:08.508786 17799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:08.511008 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 17799
I20260810 04:16:08.511082 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:08.511276 16225 external_mini_cluster.cc:1428] Setting key 9738458d768a631e15796a8714994496
I20260810 04:16:08.512639 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.501723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:58505 (local address 127.15.216.126:38241)
0810 04:16:08.501804 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.501806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.501989 (+ 183us) server_negotiation.cc:407] Connection header received
0810 04:16:08.502786 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.502788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.502804 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.502889 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.503890 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.504768 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.506612 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.506962 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.507038 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.509879 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.509881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.509881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.509889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.511123 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.511634 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.511636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.511636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.511700 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.511953 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.511954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.511954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.512075 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.512149 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.512456 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.512549 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:16:08.512782 17825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.501595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38241 (local address 127.15.216.67:58505)
0810 04:16:08.501844 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.501891 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.502566 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.502978 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.502983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.503424 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:08.503763 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.503771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.504891 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.504892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.506510 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.506514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.507076 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.507077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.507261 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.507843 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:08.507851 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:08.509779 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:08.511191 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.511199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.511202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.511547 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.511760 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.511761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.511761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.511873 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.512153 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:08.512161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:08.512294 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.512393 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.512565 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260810 04:16:08.513331 17941 heartbeater.cc:347] Connected to a master server at 127.15.216.126:38241
I20260810 04:16:08.513420 17941 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:08.513665 17941 heartbeater.cc:511] Master 127.15.216.126:38241 requested a full tablet report, sending...
I20260810 04:16:08.514119 17412 ts_manager.cc:194] Registered new tserver with Master: 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805)
I20260810 04:16:08.514532 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:58505
I20260810 04:16:08.519745 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.514547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38241 (local address 127.15.216.67:46163)
0810 04:16:08.514808 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.514818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.514897 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.515003 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.515003 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.515212 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:08.515281 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.515283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.515888 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.515889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.516397 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.516399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.516701 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.516702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.516759 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.517216 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:08.517221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:08.517492 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:08.518903 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.518903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.518904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.519064 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.519233 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.519234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.519234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.519291 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.519516 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:08.519517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:08.519553 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.519607 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.519682 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260810 04:16:08.519750 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.514618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:46163 (local address 127.15.216.126:38241)
0810 04:16:08.514663 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.514665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.514903 (+ 238us) server_negotiation.cc:407] Connection header received
0810 04:16:08.514948 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.514949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.514957 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.514996 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.515355 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.515799 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.516470 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.516637 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.516685 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.517561 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.517562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.517562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.517566 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.518828 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.519127 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.519127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.519128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.519175 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.519348 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.519349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.519349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.519452 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.519503 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.519639 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.519694 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:16:08.523010 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:08.546865 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.539709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.15.216.126:38241)
0810 04:16:08.539779 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.539781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.539790 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:08.539815 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.539816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.539825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.539877 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.540453 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.540940 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.541765 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.541949 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.542002 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.544308 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.544309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.544309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.544317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.545840 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.546180 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.546181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.546181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.546238 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.546423 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.546423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.546424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.546533 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.546615 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.546706 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.546770 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:16:08.548561 17412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:08.550309 17412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:08.557004 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.552828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:37429 (local address 127.0.0.1:53682)
0810 04:16:08.553153 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.553165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.553230 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.553418 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.553420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.553422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:08.553508 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.553511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.554526 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.554527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.555672 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.555674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.556654 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.556655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.556749 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.556777 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.556843 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.556898 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":164,"threads_started":1}
I20260810 04:16:08.557175 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.552908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.15.216.67:33805)
0810 04:16:08.553377 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.553381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.553392 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:08.553436 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.553438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.553454 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.553534 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:08.553790 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.554947 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.556186 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.556755 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.556833 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.556888 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.557016 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.557062 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":87,"threads_started":1}
I20260810 04:16:08.557485 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.552973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.15.216.65:37429)
0810 04:16:08.553140 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.553147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.553221 (+ 74us) server_negotiation.cc:407] Connection header received
0810 04:16:08.553310 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.553313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.553329 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.553401 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:08.553621 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.554431 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.555772 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.556563 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.556636 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.556691 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.557356 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.557397 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260810 04:16:08.557564 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.553348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.15.216.66:35707)
0810 04:16:08.553545 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.553549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.553558 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:08.553622 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.553624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.553639 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.553713 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:08.553905 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.554981 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.556231 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.556990 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.557204 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.557269 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.557376 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.557459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260810 04:16:08.557133 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.552820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:33805 (local address 127.0.0.1:48462)
0810 04:16:08.552993 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.553034 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.553104 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.553639 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.553640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.553643 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:08.553706 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.553708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.555046 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.555047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.556090 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.556093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.556849 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.556851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.556922 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.556926 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.556989 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.557037 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":60,"threads_started":1}
I20260810 04:16:08.557427 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.553107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:35707 (local address 127.0.0.1:37268)
0810 04:16:08.553224 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.553235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.553278 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.553790 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.553790 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.553792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:08.553826 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.553827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.555101 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.555102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.556126 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.556129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.557208 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.557208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.557264 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.557268 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.557325 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.557365 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":9,"thread_start_us":80,"threads_started":1}
I20260810 04:16:08.558451 17561 tablet_service.cc:1467] Processing CreateTablet for tablet a450bad6c4ce4ff08dc08723bc25d475 (DEFAULT_TABLE table=test-table [id=e5c4ad7643894271aae6b7dcd94c70b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:08.558625 17715 tablet_service.cc:1467] Processing CreateTablet for tablet a450bad6c4ce4ff08dc08723bc25d475 (DEFAULT_TABLE table=test-table [id=e5c4ad7643894271aae6b7dcd94c70b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:08.558858 17561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a450bad6c4ce4ff08dc08723bc25d475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.558946 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a450bad6c4ce4ff08dc08723bc25d475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.562844 17965 tablet_bootstrap.cc:492] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Bootstrap starting.
I20260810 04:16:08.563304 17867 tablet_service.cc:1467] Processing CreateTablet for tablet a450bad6c4ce4ff08dc08723bc25d475 (DEFAULT_TABLE table=test-table [id=e5c4ad7643894271aae6b7dcd94c70b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:08.563608 17965 tablet_bootstrap.cc:654] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.563694 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a450bad6c4ce4ff08dc08723bc25d475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.563956 17965 log.cc:824] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:08.564297 17964 tablet_bootstrap.cc:492] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Bootstrap starting.
I20260810 04:16:08.564759 17965 tablet_bootstrap.cc:492] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: No bootstrap required, opened a new log
I20260810 04:16:08.564836 17965 ts_tablet_manager.cc:1397] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:08.565124 17964 tablet_bootstrap.cc:654] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.565440 17964 log.cc:824] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:08.566169 17964 tablet_bootstrap.cc:492] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: No bootstrap required, opened a new log
I20260810 04:16:08.566262 17964 ts_tablet_manager.cc:1397] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:16:08.567124 17968 tablet_bootstrap.cc:492] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Bootstrap starting.
I20260810 04:16:08.567085 17965 raft_consensus.cc:348] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.567312 17965 raft_consensus.cc:374] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.567389 17965 raft_consensus.cc:729] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0263215fa9dd4626b4179d630fd270ec, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.567517 17965 consensus_queue.cc:260] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.567808 17965 ts_tablet_manager.cc:1428] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:16:08.567929 17631 heartbeater.cc:503] Master 127.15.216.126:38241 was elected leader, sending a full tablet report...
I20260810 04:16:08.568166 17968 tablet_bootstrap.cc:654] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.568192 17964 raft_consensus.cc:348] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.568367 17964 raft_consensus.cc:374] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.568455 17964 raft_consensus.cc:729] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9899dbe6205420c8eb0648991b33e27, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.568518 17968 log.cc:824] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:08.568589 17964 consensus_queue.cc:260] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.569031 17964 ts_tablet_manager.cc:1428] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:16:08.569092 17788 heartbeater.cc:503] Master 127.15.216.126:38241 was elected leader, sending a full tablet report...
I20260810 04:16:08.569483 17968 tablet_bootstrap.cc:492] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: No bootstrap required, opened a new log
I20260810 04:16:08.569551 17968 ts_tablet_manager.cc:1397] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:16:08.571173 17968 raft_consensus.cc:348] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.571292 17968 raft_consensus.cc:374] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.571314 17968 raft_consensus.cc:729] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bea558de0ef4167bb991bb324e7a6fa, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.571421 17968 consensus_queue.cc:260] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.571697 17968 ts_tablet_manager.cc:1428] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:16:08.571731 17941 heartbeater.cc:503] Master 127.15.216.126:38241 was elected leader, sending a full tablet report...
W20260810 04:16:08.598631 17789 tablet.cc:2369] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:16:08.690135 17632 tablet.cc:2369] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:08.729147 17971 raft_consensus.cc:482] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:08.729319 17971 raft_consensus.cc:504] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.729806 17971 leader_election.cc:288] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805), 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429)
I20260810 04:16:08.732777 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.730189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:36607 (local address 127.15.216.67:33805)
0810 04:16:08.730256 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.730258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.730267 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:08.730332 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.730333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.730341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.730436 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:08.730593 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.731068 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.732021 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.732402 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.732491 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.732501 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.732660 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.732700 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:16:08.732787 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.730019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:33805 (local address 127.15.216.66:36607)
0810 04:16:08.730101 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.730114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.730164 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.730441 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.730442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.730445 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:08.730517 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.730519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.731156 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.731158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.731917 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.731919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.732489 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.732490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.732580 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.732583 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.732654 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.732706 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 04:16:08.732867 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.730089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:34169 (local address 127.15.216.65:37429)
0810 04:16:08.730142 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.730144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.730258 (+ 114us) server_negotiation.cc:407] Connection header received
0810 04:16:08.730317 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.730319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.730330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.730406 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:08.730594 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.731074 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.732112 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.732471 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.732558 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.732566 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.732743 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.732786 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:16:08.732965 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.730019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:37429 (local address 127.15.216.66:34169)
0810 04:16:08.730198 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.730207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.730259 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.730417 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.730418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.730420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:08.730507 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.730514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.731166 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.731166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.732017 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.732020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.732574 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.732575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.732664 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.732668 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.732736 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.732777 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":16,"thread_start_us":97,"threads_started":1}
I20260810 04:16:08.733369 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450bad6c4ce4ff08dc08723bc25d475" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0263215fa9dd4626b4179d630fd270ec" is_pre_election: true
I20260810 04:16:08.733357 17889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450bad6c4ce4ff08dc08723bc25d475" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea558de0ef4167bb991bb324e7a6fa" is_pre_election: true
I20260810 04:16:08.733546 17889 raft_consensus.cc:2454] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 0.
I20260810 04:16:08.733531 17568 raft_consensus.cc:2454] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 0.
I20260810 04:16:08.733934 17660 leader_election.cc:302] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0263215fa9dd4626b4179d630fd270ec, c9899dbe6205420c8eb0648991b33e27; no voters:
I20260810 04:16:08.734146 17971 raft_consensus.cc:2781] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:08.734198 17971 raft_consensus.cc:482] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:08.734215 17971 raft_consensus.cc:3037] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:08.735157 17971 raft_consensus.cc:504] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.735466 17971 leader_election.cc:288] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 election: Requested vote from peers 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805), 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429)
I20260810 04:16:08.735496 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450bad6c4ce4ff08dc08723bc25d475" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0263215fa9dd4626b4179d630fd270ec"
I20260810 04:16:08.735615 17568 raft_consensus.cc:3037] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:08.736469 17568 raft_consensus.cc:2454] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 1.
I20260810 04:16:08.736622 17889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450bad6c4ce4ff08dc08723bc25d475" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea558de0ef4167bb991bb324e7a6fa"
I20260810 04:16:08.736661 17660 leader_election.cc:302] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0263215fa9dd4626b4179d630fd270ec, c9899dbe6205420c8eb0648991b33e27; no voters:
I20260810 04:16:08.736720 17889 raft_consensus.cc:3037] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:08.736799 17971 raft_consensus.cc:2781] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:08.736989 17971 raft_consensus.cc:686] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 1 LEADER]: Becoming Leader. State: Replica: c9899dbe6205420c8eb0648991b33e27, State: Running, Role: LEADER
I20260810 04:16:08.737095 17971 consensus_queue.cc:237] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } }
I20260810 04:16:08.737707 17889 raft_consensus.cc:2454] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 1.
I20260810 04:16:08.738035 17412 catalog_manager.cc:5495] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9899dbe6205420c8eb0648991b33e27 (127.15.216.66). New cstate: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } health_report { overall_health: HEALTHY } } }
W20260810 04:16:08.759583 17942 tablet.cc:2369] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:08.785372 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.782887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.15.216.66:35707)
0810 04:16:08.782962 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.782968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.782977 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:08.783014 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.783015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.783033 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.783094 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:08.783256 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.783739 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.784564 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.784754 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.784808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.785006 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:08.785096 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:08.785156 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.785221 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.785240 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 04:16:08.787456 17568 raft_consensus.cc:1261] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Refusing update from remote peer c9899dbe6205420c8eb0648991b33e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:08.787547 17889 raft_consensus.cc:1261] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Refusing update from remote peer c9899dbe6205420c8eb0648991b33e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:08.787883 17971 consensus_queue.cc:1035] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:08.788055 17974 consensus_queue.cc:1035] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:08.790747 17977 mvcc.cc:205] Tried to move back new op lower bound from 7316829670550179840 to 7316829670347644928. Current Snapshot: MvccSnapshot[applied={T|T < 7316829670550179840}]
I20260810 04:16:08.791460 17979 mvcc.cc:205] Tried to move back new op lower bound from 7316829670550179840 to 7316829670347644928. Current Snapshot: MvccSnapshot[applied={T|T < 7316829670550179840}]
I20260810 04:16:08.794528 17412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:08.794641 17412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:08.795436 17412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455: Sending DeleteTablet for 3 replicas of tablet a450bad6c4ce4ff08dc08723bc25d475
I20260810 04:16:08.795816 17867 tablet_service.cc:1514] Processing DeleteTablet for tablet a450bad6c4ce4ff08dc08723bc25d475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:08 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:48462
I20260810 04:16:08.795821 17561 tablet_service.cc:1514] Processing DeleteTablet for tablet a450bad6c4ce4ff08dc08723bc25d475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:08 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:53682
I20260810 04:16:08.796037 17412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:08.796195 17412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:08.796172 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet a450bad6c4ce4ff08dc08723bc25d475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:08 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:37268
I20260810 04:16:08.796437 17989 tablet_replica.cc:330] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: stopping tablet replica
I20260810 04:16:08.796548 17989 raft_consensus.cc:2229] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:08.796653 17989 raft_consensus.cc:2258] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:08.796769 17988 tablet_replica.cc:330] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: stopping tablet replica
I20260810 04:16:08.796823 17990 tablet_replica.cc:330] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: stopping tablet replica
I20260810 04:16:08.796929 17990 raft_consensus.cc:2229] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:08.796949 17988 raft_consensus.cc:2229] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:08.797034 17990 raft_consensus.cc:2258] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:08.797168 17988 raft_consensus.cc:2258] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:08.797900 17561 tablet_service.cc:1467] Processing CreateTablet for tablet f357f745c9a34b3ca89812a65b1e9f29 (DEFAULT_TABLE table=test-table [id=14b2075f86b64daea4f22a68ece71d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:08.798038 17561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f357f745c9a34b3ca89812a65b1e9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.798010 17715 tablet_service.cc:1467] Processing CreateTablet for tablet f357f745c9a34b3ca89812a65b1e9f29 (DEFAULT_TABLE table=test-table [id=14b2075f86b64daea4f22a68ece71d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:08.798180 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f357f745c9a34b3ca89812a65b1e9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.798290 17867 tablet_service.cc:1467] Processing CreateTablet for tablet f357f745c9a34b3ca89812a65b1e9f29 (DEFAULT_TABLE table=test-table [id=14b2075f86b64daea4f22a68ece71d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:08.798405 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f357f745c9a34b3ca89812a65b1e9f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.799427 17965 tablet_bootstrap.cc:492] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: Bootstrap starting.
I20260810 04:16:08.799810 17965 tablet_bootstrap.cc:654] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.799858 17968 tablet_bootstrap.cc:492] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: Bootstrap starting.
I20260810 04:16:08.799932 17990 ts_tablet_manager.cc:1905] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:08.800264 17964 tablet_bootstrap.cc:492] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: Bootstrap starting.
I20260810 04:16:08.800292 17988 ts_tablet_manager.cc:1905] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:08.800341 17968 tablet_bootstrap.cc:654] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.800734 17964 tablet_bootstrap.cc:654] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.801776 17989 ts_tablet_manager.cc:1905] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:08.801832 17964 tablet_bootstrap.cc:492] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: No bootstrap required, opened a new log
I20260810 04:16:08.801882 17964 ts_tablet_manager.cc:1397] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:16:08.802059 17964 raft_consensus.cc:348] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.802148 17964 raft_consensus.cc:374] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.802179 17964 raft_consensus.cc:729] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9899dbe6205420c8eb0648991b33e27, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.802227 17964 consensus_queue.cc:260] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.802270 17988 ts_tablet_manager.cc:1918] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:08.802333 17964 ts_tablet_manager.cc:1428] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.802428 17988 log.cc:1196] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/a450bad6c4ce4ff08dc08723bc25d475
I20260810 04:16:08.802736 17968 tablet_bootstrap.cc:492] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: No bootstrap required, opened a new log
I20260810 04:16:08.802795 17968 ts_tablet_manager.cc:1397] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:16:08.802796 17988 ts_tablet_manager.cc:1939] T a450bad6c4ce4ff08dc08723bc25d475 P 0263215fa9dd4626b4179d630fd270ec: Deleting consensus metadata
I20260810 04:16:08.802973 17968 raft_consensus.cc:348] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.803056 17968 raft_consensus.cc:374] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.803082 17968 raft_consensus.cc:729] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bea558de0ef4167bb991bb324e7a6fa, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.803126 17968 consensus_queue.cc:260] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.803216 17968 ts_tablet_manager.cc:1428] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:08.803227 17397 catalog_manager.cc:4841] TS 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429): tablet a450bad6c4ce4ff08dc08723bc25d475 (table test-table [id=e5c4ad7643894271aae6b7dcd94c70b4]) successfully deleted
I20260810 04:16:08.803874 17990 ts_tablet_manager.cc:1918] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:08.804033 17990 log.cc:1196] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/a450bad6c4ce4ff08dc08723bc25d475
I20260810 04:16:08.804371 17989 ts_tablet_manager.cc:1918] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:08.804415 17990 ts_tablet_manager.cc:1939] T a450bad6c4ce4ff08dc08723bc25d475 P c9899dbe6205420c8eb0648991b33e27: Deleting consensus metadata
I20260810 04:16:08.804522 17989 log.cc:1196] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/a450bad6c4ce4ff08dc08723bc25d475
I20260810 04:16:08.804833 17397 catalog_manager.cc:4841] TS c9899dbe6205420c8eb0648991b33e27 (127.15.216.66:35707): tablet a450bad6c4ce4ff08dc08723bc25d475 (table test-table [id=e5c4ad7643894271aae6b7dcd94c70b4]) successfully deleted
I20260810 04:16:08.804863 17989 ts_tablet_manager.cc:1939] T a450bad6c4ce4ff08dc08723bc25d475 P 7bea558de0ef4167bb991bb324e7a6fa: Deleting consensus metadata
I20260810 04:16:08.805066 17965 tablet_bootstrap.cc:492] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: No bootstrap required, opened a new log
I20260810 04:16:08.805122 17965 ts_tablet_manager.cc:1397] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260810 04:16:08.805255 17398 catalog_manager.cc:4841] TS 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805): tablet a450bad6c4ce4ff08dc08723bc25d475 (table test-table [id=e5c4ad7643894271aae6b7dcd94c70b4]) successfully deleted
I20260810 04:16:08.805457 17965 raft_consensus.cc:348] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.805531 17965 raft_consensus.cc:374] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.805546 17965 raft_consensus.cc:729] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0263215fa9dd4626b4179d630fd270ec, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.805635 17965 consensus_queue.cc:260] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.805758 17965 ts_tablet_manager.cc:1428] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:08.852823 17974 raft_consensus.cc:482] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:08.852938 17974 raft_consensus.cc:504] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.853096 17974 leader_election.cc:288] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429), 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805)
I20260810 04:16:08.853354 17889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f357f745c9a34b3ca89812a65b1e9f29" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea558de0ef4167bb991bb324e7a6fa" is_pre_election: true
I20260810 04:16:08.853366 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f357f745c9a34b3ca89812a65b1e9f29" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0263215fa9dd4626b4179d630fd270ec" is_pre_election: true
I20260810 04:16:08.853461 17889 raft_consensus.cc:2454] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 0.
I20260810 04:16:08.853492 17568 raft_consensus.cc:2454] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 0.
I20260810 04:16:08.853643 17661 leader_election.cc:302] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bea558de0ef4167bb991bb324e7a6fa, c9899dbe6205420c8eb0648991b33e27; no voters:
I20260810 04:16:08.853760 17974 raft_consensus.cc:2781] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:08.853796 17974 raft_consensus.cc:482] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:08.853814 17974 raft_consensus.cc:3037] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:08.854645 17974 raft_consensus.cc:504] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.854794 17974 leader_election.cc:288] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 election: Requested vote from peers 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429), 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805)
I20260810 04:16:08.855005 17889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f357f745c9a34b3ca89812a65b1e9f29" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea558de0ef4167bb991bb324e7a6fa"
I20260810 04:16:08.855011 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f357f745c9a34b3ca89812a65b1e9f29" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0263215fa9dd4626b4179d630fd270ec"
I20260810 04:16:08.855093 17889 raft_consensus.cc:3037] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:08.855093 17568 raft_consensus.cc:3037] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:08.855873 17568 raft_consensus.cc:2454] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 1.
I20260810 04:16:08.855873 17889 raft_consensus.cc:2454] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 1.
I20260810 04:16:08.856060 17660 leader_election.cc:302] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0263215fa9dd4626b4179d630fd270ec, c9899dbe6205420c8eb0648991b33e27; no voters:
I20260810 04:16:08.856185 17974 raft_consensus.cc:2781] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:08.856235 17974 raft_consensus.cc:686] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 1 LEADER]: Becoming Leader. State: Replica: c9899dbe6205420c8eb0648991b33e27, State: Running, Role: LEADER
I20260810 04:16:08.856284 17974 consensus_queue.cc:237] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.856870 17412 catalog_manager.cc:5495] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9899dbe6205420c8eb0648991b33e27 (127.15.216.66). New cstate: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } health_report { overall_health: UNKNOWN } } }
Aug 10 04:16:08 dist-test-slave-g350 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335367, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:08.902014 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.894448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.15.216.126:38241)
0810 04:16:08.894544 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:08.894547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:08.894694 (+ 147us) server_negotiation.cc:407] Connection header received
0810 04:16:08.894769 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:08.894770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:08.894780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:08.894865 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:08.895356 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.895959 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.896774 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.896969 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.897040 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.899346 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:08.899348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:08.899349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:08.899356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:08.900765 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.901167 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.901168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.901169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.901225 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:08.901438 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:08.901439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:08.901440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:08.901550 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:08.901647 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:08.901813 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 04:16:08.901906 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 04:16:08.902096 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:08.894391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38241 (local address 127.0.0.1:45686)
0810 04:16:08.894619 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:08.894629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:08.894712 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:08.894872 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:08.894873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:08.895221 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:08.895279 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.895282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.896041 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.896042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.896685 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:08.896688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:08.897052 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:08.897053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:08.897107 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:08.897690 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:08.897696 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:08.899253 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:08.900871 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.900880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.900882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.901074 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.901297 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:08.901298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:08.901298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:08.901361 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:08.901650 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:08.901658 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:08.901704 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:16:08.901788 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 04:16:08.901961 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":73,"thread_start_us":114,"threads_started":1}
I20260810 04:16:08.904686 17412 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:45686:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:16:08.905043 17412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:08.906618 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 0e66b9aac0c14e188a00945f99372899 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33c02a5cc9364a72ad10294a8f78e921]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:08.906596 17561 tablet_service.cc:1467] Processing CreateTablet for tablet 0e66b9aac0c14e188a00945f99372899 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33c02a5cc9364a72ad10294a8f78e921]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:08.906852 17561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e66b9aac0c14e188a00945f99372899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.906868 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e66b9aac0c14e188a00945f99372899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.908195 17964 tablet_bootstrap.cc:492] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: Bootstrap starting.
I20260810 04:16:08.908427 17867 tablet_service.cc:1467] Processing CreateTablet for tablet 0e66b9aac0c14e188a00945f99372899 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33c02a5cc9364a72ad10294a8f78e921]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:08.908564 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e66b9aac0c14e188a00945f99372899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:08.908636 17964 tablet_bootstrap.cc:654] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.909256 17964 tablet_bootstrap.cc:492] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: No bootstrap required, opened a new log
I20260810 04:16:08.909325 17964 ts_tablet_manager.cc:1397] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:08.909437 17964 raft_consensus.cc:348] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.909523 17964 raft_consensus.cc:374] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.909554 17964 raft_consensus.cc:729] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9899dbe6205420c8eb0648991b33e27, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.909597 17964 consensus_queue.cc:260] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.909718 17964 ts_tablet_manager.cc:1428] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:16:08.909832 17974 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:08.909919 17974 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:08.910104 17965 tablet_bootstrap.cc:492] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: Bootstrap starting.
I20260810 04:16:08.910610 17965 tablet_bootstrap.cc:654] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.911015 17968 tablet_bootstrap.cc:492] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: Bootstrap starting.
I20260810 04:16:08.911329 17965 tablet_bootstrap.cc:492] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: No bootstrap required, opened a new log
I20260810 04:16:08.911397 17965 ts_tablet_manager.cc:1397] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:08.911507 17968 tablet_bootstrap.cc:654] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:08.911577 17965 raft_consensus.cc:348] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.911655 17965 raft_consensus.cc:374] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.911682 17965 raft_consensus.cc:729] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0263215fa9dd4626b4179d630fd270ec, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.911732 17965 consensus_queue.cc:260] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.911839 17965 ts_tablet_manager.cc:1428] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:16:08.911988 17969 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:08.912067 17969 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:08.912528 17968 tablet_bootstrap.cc:492] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: No bootstrap required, opened a new log
I20260810 04:16:08.912580 17968 ts_tablet_manager.cc:1397] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:08.912760 17968 raft_consensus.cc:348] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.912900 17968 raft_consensus.cc:374] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:08.912935 17968 raft_consensus.cc:729] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bea558de0ef4167bb991bb324e7a6fa, State: Initialized, Role: FOLLOWER
I20260810 04:16:08.912987 17968 consensus_queue.cc:260] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:08.913079 17968 ts_tablet_manager.cc:1428] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:08.913215 17972 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:08.913302 17972 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:09.002331 17974 raft_consensus.cc:482] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:09.002462 17974 raft_consensus.cc:504] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:09.002691 17974 leader_election.cc:288] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429), 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805)
I20260810 04:16:09.002879 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e66b9aac0c14e188a00945f99372899" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0263215fa9dd4626b4179d630fd270ec" is_pre_election: true
I20260810 04:16:09.002954 17889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e66b9aac0c14e188a00945f99372899" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea558de0ef4167bb991bb324e7a6fa" is_pre_election: true
I20260810 04:16:09.003029 17568 raft_consensus.cc:2454] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 0.
I20260810 04:16:09.003031 17889 raft_consensus.cc:2454] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 0.
I20260810 04:16:09.003203 17661 leader_election.cc:302] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bea558de0ef4167bb991bb324e7a6fa, c9899dbe6205420c8eb0648991b33e27; no voters:
I20260810 04:16:09.003312 17974 raft_consensus.cc:2781] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:09.003357 17974 raft_consensus.cc:482] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:09.003388 17974 raft_consensus.cc:3037] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:09.004078 17974 raft_consensus.cc:504] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:09.004230 17974 leader_election.cc:288] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 election: Requested vote from peers 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429), 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805)
I20260810 04:16:09.004380 17889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e66b9aac0c14e188a00945f99372899" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bea558de0ef4167bb991bb324e7a6fa"
I20260810 04:16:09.004383 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e66b9aac0c14e188a00945f99372899" candidate_uuid: "c9899dbe6205420c8eb0648991b33e27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0263215fa9dd4626b4179d630fd270ec"
I20260810 04:16:09.004459 17889 raft_consensus.cc:3037] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:09.004477 17568 raft_consensus.cc:3037] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:09.005059 17889 raft_consensus.cc:2454] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 1.
I20260810 04:16:09.005141 17568 raft_consensus.cc:2454] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9899dbe6205420c8eb0648991b33e27 in term 1.
I20260810 04:16:09.005235 17661 leader_election.cc:302] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bea558de0ef4167bb991bb324e7a6fa, c9899dbe6205420c8eb0648991b33e27; no voters:
I20260810 04:16:09.005329 17974 raft_consensus.cc:2781] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:09.005384 17974 raft_consensus.cc:686] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [term 1 LEADER]: Becoming Leader. State: Replica: c9899dbe6205420c8eb0648991b33e27, State: Running, Role: LEADER
I20260810 04:16:09.005440 17974 consensus_queue.cc:237] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } }
I20260810 04:16:09.005733 17971 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: TxnStatusTablet state changed. Reason: New leader c9899dbe6205420c8eb0648991b33e27. Latest consensus state: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:09.005824 17971 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:09.006007 18007 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:16:09.006142 18007 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:16:09.006064 17413 catalog_manager.cc:5495] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9899dbe6205420c8eb0648991b33e27 (127.15.216.66). New cstate: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } health_report { overall_health: UNKNOWN } } }
I20260810 04:16:09.031265 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:09.028570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:35707 (local address 127.0.0.1:37272)
0810 04:16:09.028646 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:09.028660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:09.028715 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:09.028860 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:09.028861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:09.028862 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:09.028914 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:09.028917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.029834 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.029834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:09.030408 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:09.030411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.030879 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.030880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:09.030945 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:09.030961 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:09.031137 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:09.031140 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:09.031171 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 04:16:09.031205 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:16:09.031286 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:09.028637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.15.216.66:35707)
0810 04:16:09.028683 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:09.028685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:09.028695 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:09.028775 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:09.028775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:09.028786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:09.028857 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:09.028953 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.029756 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.030497 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.030799 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.030878 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:09.031046 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:09.031079 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:09.031137 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:09.031174 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 04:16:09.031198 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:16:09.032675 17568 raft_consensus.cc:1261] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Refusing update from remote peer c9899dbe6205420c8eb0648991b33e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:09.032708 17889 raft_consensus.cc:1261] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Refusing update from remote peer c9899dbe6205420c8eb0648991b33e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:09.032954 17971 consensus_queue.cc:1035] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:09.033155 17971 consensus_queue.cc:1035] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:09.033577 17969 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: TxnStatusTablet state changed. Reason: New leader c9899dbe6205420c8eb0648991b33e27. Latest consensus state: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:09.033704 17969 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:09.033653 17972 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: TxnStatusTablet state changed. Reason: New leader c9899dbe6205420c8eb0648991b33e27. Latest consensus state: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:09.033741 17972 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:09.034471 17971 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:09.034547 17971 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:09.034590 17971 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:09.034624 17971 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P c9899dbe6205420c8eb0648991b33e27: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:09.035025 17978 mvcc.cc:205] Tried to move back new op lower bound from 7316829671555678208 to 7316829671446626304. Current Snapshot: MvccSnapshot[applied={T|T < 7316829671555678208}]
I20260810 04:16:09.035310 17972 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:09.035382 17972 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P 7bea558de0ef4167bb991bb324e7a6fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:09.035326 17969 tablet_replica.cc:439] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9899dbe6205420c8eb0648991b33e27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 } } peers { permanent_uuid: "c9899dbe6205420c8eb0648991b33e27" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 35707 } } peers { permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 } } }
I20260810 04:16:09.035409 17969 tablet_replica.cc:442] T 0e66b9aac0c14e188a00945f99372899 P 0263215fa9dd4626b4179d630fd270ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:09.042062 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:09.039416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:35707 (local address 127.15.216.66:47137)
0810 04:16:09.039577 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:09.039587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:09.039647 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:09.039766 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:09.039767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:09.039768 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:09.039811 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:09.039812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.040627 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.040628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:09.041197 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:09.041198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.041668 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.041669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:09.041720 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:09.041735 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:09.041908 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:09.041909 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:16:09.041944 (+ 35us) client_negotiation.cc:240] Negotiation successful
0810 04:16:09.041982 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260810 04:16:09.042061 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:09.039450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:47137 (local address 127.15.216.66:35707)
0810 04:16:09.039515 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:09.039517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:09.039656 (+ 139us) server_negotiation.cc:407] Connection header received
0810 04:16:09.039696 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:09.039697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:09.039708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:09.039765 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:09.039848 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.040553 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.041283 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.041573 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.041666 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:09.041815 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:09.041846 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:09.041899 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:09.041947 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 04:16:09.041965 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:16:09.044447 17889 raft_consensus.cc:1261] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Refusing update from remote peer c9899dbe6205420c8eb0648991b33e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:09.044474 17568 raft_consensus.cc:1261] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Refusing update from remote peer c9899dbe6205420c8eb0648991b33e27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:09.044627 17985 consensus_queue.cc:1035] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bea558de0ef4167bb991bb324e7a6fa" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 33805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:09.044710 17974 consensus_queue.cc:1035] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0263215fa9dd4626b4179d630fd270ec" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 37429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:09.056082 17413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:09.056191 17413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:09.056874 17413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfac258c299f422baaf14d5eb4001455: Sending DeleteTablet for 3 replicas of tablet f357f745c9a34b3ca89812a65b1e9f29
I20260810 04:16:09.057101 17561 tablet_service.cc:1514] Processing DeleteTablet for tablet f357f745c9a34b3ca89812a65b1e9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:09 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:53682
I20260810 04:16:09.057139 17713 tablet_service.cc:1514] Processing DeleteTablet for tablet f357f745c9a34b3ca89812a65b1e9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:09 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:37268
I20260810 04:16:09.057260 17990 tablet_replica.cc:330] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: stopping tablet replica
I20260810 04:16:09.057256 17988 tablet_replica.cc:330] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: stopping tablet replica
I20260810 04:16:09.057256 17867 tablet_service.cc:1514] Processing DeleteTablet for tablet f357f745c9a34b3ca89812a65b1e9f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:09 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:48462
I20260810 04:16:09.057312 17990 raft_consensus.cc:2229] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:09.057317 17988 raft_consensus.cc:2229] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:09.057359 17988 raft_consensus.cc:2258] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:09.057366 17990 raft_consensus.cc:2258] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:09.057428 17989 tablet_replica.cc:330] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: stopping tablet replica
I20260810 04:16:09.057477 17989 raft_consensus.cc:2229] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:09.057511 17989 raft_consensus.cc:2258] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:09.057694 17990 ts_tablet_manager.cc:1905] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:09.058229 17989 ts_tablet_manager.cc:1905] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:09.058229 17988 ts_tablet_manager.cc:1905] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:09.059569 17990 ts_tablet_manager.cc:1918] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:09.059624 17990 log.cc:1196] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/f357f745c9a34b3ca89812a65b1e9f29
I20260810 04:16:09.059733 17989 ts_tablet_manager.cc:1918] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:09.059732 17988 ts_tablet_manager.cc:1918] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:09.059787 17989 log.cc:1196] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/f357f745c9a34b3ca89812a65b1e9f29
I20260810 04:16:09.059787 17988 log.cc:1196] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/f357f745c9a34b3ca89812a65b1e9f29
I20260810 04:16:09.060012 17988 ts_tablet_manager.cc:1939] T f357f745c9a34b3ca89812a65b1e9f29 P 0263215fa9dd4626b4179d630fd270ec: Deleting consensus metadata
I20260810 04:16:09.060012 17989 ts_tablet_manager.cc:1939] T f357f745c9a34b3ca89812a65b1e9f29 P 7bea558de0ef4167bb991bb324e7a6fa: Deleting consensus metadata
I20260810 04:16:09.060017 17990 ts_tablet_manager.cc:1939] T f357f745c9a34b3ca89812a65b1e9f29 P c9899dbe6205420c8eb0648991b33e27: Deleting consensus metadata
I20260810 04:16:09.060379 17397 catalog_manager.cc:4841] TS c9899dbe6205420c8eb0648991b33e27 (127.15.216.66:35707): tablet f357f745c9a34b3ca89812a65b1e9f29 (table test-table [id=14b2075f86b64daea4f22a68ece71d02]) successfully deleted
I20260810 04:16:09.060675 17398 catalog_manager.cc:4841] TS 7bea558de0ef4167bb991bb324e7a6fa (127.15.216.67:33805): tablet f357f745c9a34b3ca89812a65b1e9f29 (table test-table [id=14b2075f86b64daea4f22a68ece71d02]) successfully deleted
I20260810 04:16:09.060775 17397 catalog_manager.cc:4841] TS 0263215fa9dd4626b4179d630fd270ec (127.15.216.65:37429): tablet f357f745c9a34b3ca89812a65b1e9f29 (table test-table [id=14b2075f86b64daea4f22a68ece71d02]) successfully deleted
Aug 10 04:16:09 dist-test-slave-g350 krb5kdc[17362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335368, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:09.070319 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:09.060420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.15.216.65:37429)
0810 04:16:09.060471 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:09.060474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:09.061779 (+ 1305us) server_negotiation.cc:407] Connection header received
0810 04:16:09.061847 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:09.061848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:09.061858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:09.061910 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:09.062501 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.062973 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.063691 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.063901 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.063980 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:09.067253 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:09.067262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:09.067263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:09.067270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:09.069088 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:09.069423 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:09.069427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:09.069428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:09.069495 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:09.069744 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:09.069745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:09.069746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:09.069925 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:09.070079 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:09.070133 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:16:09.070218 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260810 04:16:09.070611 17625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53686
I20260810 04:16:09.078328 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:09.071664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.15.216.126:38241)
0810 04:16:09.071726 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:09.071728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:09.071823 (+ 95us) server_negotiation.cc:407] Connection header received
0810 04:16:09.071914 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:09.071915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:09.071926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:09.071972 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:09.072562 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.073235 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.074061 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:09.074370 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:09.074439 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:09.075644 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:09.075646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:09.075646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:09.075653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:09.077139 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:09.077505 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:09.077507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:09.077507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:09.077571 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:09.077806 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:09.077807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:09.077808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:09.077933 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:09.078042 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:09.078143 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 04:16:09.078221 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 04:16:09.078596 17413 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45692
I20260810 04:16:09.079276 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 17492
I20260810 04:16:09.091747 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 17640
I20260810 04:16:09.100883 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 17799
I20260810 04:16:09.107394 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 17377
2026-08-10T04:16:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2486 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:09 dist-test-slave-g350 krb5kdc[18024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:09 dist-test-slave-g350 krb5kdc[18024](info): set up 2 sockets
Aug 10 04:16:09 dist-test-slave-g350 krb5kdc[18024](info): commencing operation
krb5kdc: starting...
W20260810 04:16:10.218374 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:10 dist-test-slave-g350 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:10.354230 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36009
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:10.437865 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:10.438067 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:10.438089 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:10.439316 18038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:10.439356 18038 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:16:10.439370 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:10.439383 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:10.439402 18038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:10.439451 18038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:10.440940 18038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:36009
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:10.441197 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:10.441454 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:10.446658 18047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:10.446672 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:10.446686 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:10.446969 18038 server_base.cc:1034] running on GCE node
I20260810 04:16:10.447235 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:10.447546 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:10.448729 18038 hybrid_clock.cc:648] HybridClock initialized: now 1786335370448702 us; error 60 us; skew 500 ppm
Aug 10 04:16:10 dist-test-slave-g350 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:10.451465 18038 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:10.451924 18038 webserver.cc:460] Webserver started at http://127.15.216.126:44717/ using document root <none> and password file <none>
I20260810 04:16:10.452149 18038 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:10.452219 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:10.452355 18038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:10.453398 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "998c7f08b19c4ddd81243cce7baef069"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "bc78292f815280b8f5a588867561636b"
server_key_iv: "a7750cc8730d552908f48b7d84a6a329"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.453807 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "998c7f08b19c4ddd81243cce7baef069"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "bc78292f815280b8f5a588867561636b"
server_key_iv: "a7750cc8730d552908f48b7d84a6a329"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.455157 18038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:10.455865 18055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:10.456073 18038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:16:10.456184 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "998c7f08b19c4ddd81243cce7baef069"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "bc78292f815280b8f5a588867561636b"
server_key_iv: "a7750cc8730d552908f48b7d84a6a329"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.456264 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:10.480587 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:10.482720 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:10.482908 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:10.487608 18038 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:36009
I20260810 04:16:10.487720 18129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:36009 every 8 connection(s)
I20260810 04:16:10.488026 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:10.488730 18130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:10.489876 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 18038
I20260810 04:16:10.489948 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:10.490126 16225 external_mini_cluster.cc:1428] Setting key 96520305ab78aa92df8fa2ac5f4b4941
I20260810 04:16:10.491365 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Bootstrap starting.
I20260810 04:16:10.491990 18130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:10.492314 18130 log.cc:824] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:10.493037 18130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: No bootstrap required, opened a new log
I20260810 04:16:10.495529 18130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998c7f08b19c4ddd81243cce7baef069" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36009 } }
I20260810 04:16:10.495725 18130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:10.495770 18130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998c7f08b19c4ddd81243cce7baef069, State: Initialized, Role: FOLLOWER
I20260810 04:16:10.495946 18130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998c7f08b19c4ddd81243cce7baef069" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36009 } }
I20260810 04:16:10.496050 18130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:10.496088 18130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:10.496129 18130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:10.496767 18130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998c7f08b19c4ddd81243cce7baef069" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36009 } }
I20260810 04:16:10.496863 18130 leader_election.cc:302] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 998c7f08b19c4ddd81243cce7baef069; no voters:
Aug 10 04:16:10 dist-test-slave-g350 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:10.497087 18130 leader_election.cc:288] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:10.497267 18137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:10.497378 18130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:10.497408 18137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [term 1 LEADER]: Becoming Leader. State: Replica: 998c7f08b19c4ddd81243cce7baef069, State: Running, Role: LEADER
I20260810 04:16:10.497686 18137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998c7f08b19c4ddd81243cce7baef069" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36009 } }
I20260810 04:16:10.498018 18137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 998c7f08b19c4ddd81243cce7baef069. Latest consensus state: current_term: 1 leader_uuid: "998c7f08b19c4ddd81243cce7baef069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998c7f08b19c4ddd81243cce7baef069" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36009 } } }
I20260810 04:16:10.498184 18137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:10.498184 18138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "998c7f08b19c4ddd81243cce7baef069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998c7f08b19c4ddd81243cce7baef069" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36009 } } }
I20260810 04:16:10.498407 18138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:10.498508 18145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:10.499083 18145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:10.501426 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.490716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.15.216.126:36009)
0810 04:16:10.490906 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:10.490914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:10.490938 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:16:10.491328 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:10.491334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:10.491376 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:10.491525 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:10.492149 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.493110 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.494275 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.494694 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.494793 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.497572 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:10.497576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:10.497576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:10.497588 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:10.499700 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.500166 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.500169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.500170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.500275 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.500515 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.500516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.500516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.500725 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:10.500835 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:10.501006 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:16:10.501119 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260810 04:16:10.502641 18145 catalog_manager.cc:1347] Generated new cluster ID: 48940e32648f439c8b81215d90538b84
I20260810 04:16:10.502696 18145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:10.521042 18145 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:10.521884 18145 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:10.526713 18145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Generated new TSK 0
I20260810 04:16:10.526904 18145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:10.557670 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36009
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:10.641182 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:10.641434 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:10.641477 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:10.642629 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:10.642707 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:10.642745 18159 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:16:10.642795 18159 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:10.642856 18159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:10.644325 18159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:10.644588 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:10.644845 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:10.650168 18165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:10.650207 18169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:10.650177 18166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:10.650460 18159 server_base.cc:1034] running on GCE node
I20260810 04:16:10.650702 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:10.651044 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:10.652215 18159 hybrid_clock.cc:648] HybridClock initialized: now 1786335370652190 us; error 40 us; skew 500 ppm
Aug 10 04:16:10 dist-test-slave-g350 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:10.654909 18159 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:10.655385 18159 webserver.cc:460] Webserver started at http://127.15.216.65:41793/ using document root <none> and password file <none>
I20260810 04:16:10.655571 18159 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:10.655612 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:10.655702 18159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:10.656776 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "97f5aa45ed7744f19f869ec1a1ad8092"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "c42f4393babbc8b0c703585d45b19be1"
server_key_iv: "3065484f6662d3ab64483fcac8dbf597"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.657080 18159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "97f5aa45ed7744f19f869ec1a1ad8092"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "c42f4393babbc8b0c703585d45b19be1"
server_key_iv: "3065484f6662d3ab64483fcac8dbf597"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.658897 18159 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 04:16:10.660023 18176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:10.660362 18159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:10.660503 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "97f5aa45ed7744f19f869ec1a1ad8092"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "c42f4393babbc8b0c703585d45b19be1"
server_key_iv: "3065484f6662d3ab64483fcac8dbf597"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.660604 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:10.691403 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:10.694573 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:10.694803 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:10.695556 18159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:10.695605 18159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:10.695636 18159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:10.695659 18159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:16:10 dist-test-slave-g350 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:10.703953 18159 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:33881
I20260810 04:16:10.704043 18301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:33881 every 8 connection(s)
I20260810 04:16:10.704392 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:10.707980 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.696174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:44077 (local address 127.15.216.126:36009)
0810 04:16:10.696243 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:10.696245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:10.696495 (+ 250us) server_negotiation.cc:407] Connection header received
0810 04:16:10.697212 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:10.697214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:10.697229 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:10.697295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:10.698134 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.698918 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.700453 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.700787 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.700837 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.704041 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:10.704043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:10.704043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:10.704052 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:10.706260 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.706828 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.706829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.706829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.706890 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.707237 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.707238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.707239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.707393 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:10.707453 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:10.707829 (+ 376us) server_negotiation.cc:299] Negotiation successful
0810 04:16:10.707915 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:16:10.708287 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.696186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36009 (local address 127.15.216.65:44077)
0810 04:16:10.696375 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:10.696417 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:10.697070 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:10.697416 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:10.697421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:10.697749 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:10.698019 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.698025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.699080 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.699081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.700347 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.700350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.700950 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.700951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.701085 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.702058 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:10.702068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:10.703914 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:10.706350 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.706358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.706361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.706720 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.706978 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.706979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.706980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.707157 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.707468 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:10.707475 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:10.707667 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 04:16:10.707803 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 04:16:10.708010 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":156,"threads_started":1}
I20260810 04:16:10.708863 18302 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36009
I20260810 04:16:10.708976 18302 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:10.709168 18302 heartbeater.cc:511] Master 127.15.216.126:36009 requested a full tablet report, sending...
I20260810 04:16:10.709811 18075 ts_manager.cc:194] Registered new tserver with Master: 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881)
I20260810 04:16:10.710747 18075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:44077
I20260810 04:16:10.713835 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 18159
I20260810 04:16:10.713953 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:10.714177 16225 external_mini_cluster.cc:1428] Setting key ee0569b99091e29aed2972776f9bb1cb
I20260810 04:16:10.719292 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.710126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36009 (local address 127.15.216.65:59401)
0810 04:16:10.710265 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:10.710276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:10.710337 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:10.710509 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:10.710510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:10.710719 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:10.710789 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.710792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.711677 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.711678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.712611 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.712614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.713011 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.713012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.713093 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.713668 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:10.713673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:10.714277 (+ 604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:10.718159 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.718160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.718160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.718431 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.718647 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.718647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.718648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.718713 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.718960 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:10.718961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:10.719012 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 04:16:10.719078 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:16:10.719196 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260810 04:16:10.719290 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.710189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:59401 (local address 127.15.216.126:36009)
0810 04:16:10.710227 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:10.710229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:10.710338 (+ 109us) server_negotiation.cc:407] Connection header received
0810 04:16:10.710391 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:10.710392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:10.710402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:10.710448 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:10.711003 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.711490 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.712719 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.712928 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.713006 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.714351 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:10.714352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:10.714353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:10.714361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:10.718056 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.718506 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.718507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.718508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.718565 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.718764 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.718765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.718766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.718879 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:10.718954 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:10.719112 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 04:16:10.719192 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:10.746254 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36009
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:10.831084 18312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:10.831359 18312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:10.831391 18312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:10.832554 18312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:10.832597 18312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:10.832617 18312 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:16:10.832630 18312 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:10.832679 18312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:10.834167 18312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:10.834420 18312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:10.834664 18312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:10.840091 18317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:10.840111 18320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:10.840066 18318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:10.840332 18312 server_base.cc:1034] running on GCE node
I20260810 04:16:10.840529 18312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:10.840790 18312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:10.841953 18312 hybrid_clock.cc:648] HybridClock initialized: now 1786335370841951 us; error 40 us; skew 500 ppm
Aug 10 04:16:10 dist-test-slave-g350 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:10.844424 18312 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:10.844910 18312 webserver.cc:460] Webserver started at http://127.15.216.66:35051/ using document root <none> and password file <none>
I20260810 04:16:10.845157 18312 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:10.845228 18312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:10.845367 18312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:10.846549 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "6a7d16f92878430ca0bd54598124613c"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "a58528db251b7d522e1f345b3b77afd5"
server_key_iv: "0c60fbc49d21419ba5d2ede6d4854e5b"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.846931 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "6a7d16f92878430ca0bd54598124613c"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "a58528db251b7d522e1f345b3b77afd5"
server_key_iv: "0c60fbc49d21419ba5d2ede6d4854e5b"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.848598 18312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:10.849421 18327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:10.849762 18312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:10.849870 18312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "6a7d16f92878430ca0bd54598124613c"
format_stamp: "Formatted at 2026-08-10 04:16:10 on dist-test-slave-g350"
server_key: "a58528db251b7d522e1f345b3b77afd5"
server_key_iv: "0c60fbc49d21419ba5d2ede6d4854e5b"
server_key_version: "encryptionkey@0"
I20260810 04:16:10.849957 18312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:10.861934 18312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:10.863997 18312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:10.864171 18312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:10.864799 18312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:10.864863 18312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:10.864903 18312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:10.864931 18312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:16:10 dist-test-slave-g350 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:10.872985 18312 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:38125
I20260810 04:16:10.873104 18447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:38125 every 8 connection(s)
I20260810 04:16:10.873425 18312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:10.875820 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.865320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:45839 (local address 127.15.216.126:36009)
0810 04:16:10.865372 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:10.865374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:10.865644 (+ 270us) server_negotiation.cc:407] Connection header received
0810 04:16:10.866392 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:10.866393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:10.866410 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:10.866450 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:10.867374 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.868094 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.869274 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.869515 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.869642 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.872574 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:10.872575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:10.872576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:10.872586 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:10.874431 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.874859 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.874860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.874861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.874928 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.875176 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.875177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.875177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.875371 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:10.875436 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:10.875670 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 04:16:10.875751 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:16:10.875973 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.865274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36009 (local address 127.15.216.66:45839)
0810 04:16:10.865492 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:10.865530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:10.866149 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:10.866669 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:10.866675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:10.867023 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:10.867240 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.867245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.868215 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.868216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.869123 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.869126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.869661 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.869663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.869861 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.870511 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:10.870520 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:10.872432 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:10.874517 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.874523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.874524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.874745 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.875009 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.875009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.875010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.875112 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.875437 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:10.875442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:10.875554 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 04:16:10.875637 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:16:10.875760 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":116,"threads_started":1}
I20260810 04:16:10.876614 18448 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36009
I20260810 04:16:10.876703 18448 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:10.876909 18448 heartbeater.cc:511] Master 127.15.216.126:36009 requested a full tablet report, sending...
I20260810 04:16:10.877333 18075 ts_manager.cc:194] Registered new tserver with Master: 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125)
I20260810 04:16:10.877936 18075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:45839
I20260810 04:16:10.882121 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 18312
I20260810 04:16:10.882252 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:10.882516 16225 external_mini_cluster.cc:1428] Setting key 8faf02f10f31577804351e71115d85ff
I20260810 04:16:10.884315 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.877814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36009 (local address 127.15.216.66:46165)
0810 04:16:10.877983 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:10.877996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:10.878056 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:10.878293 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:10.878295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:10.878522 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:10.878599 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.878601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.879269 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.879270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.879861 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:10.879863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.880331 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.880332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:10.880407 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.880812 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:10.880817 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:10.881157 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:10.883189 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.883191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.883191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.883400 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.883625 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:10.883626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:10.883626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:10.883682 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:10.884074 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:10.884075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:10.884113 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 04:16:10.884158 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 04:16:10.884239 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260810 04:16:10.884470 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:10.877937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:46165 (local address 127.15.216.126:36009)
0810 04:16:10.878143 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:10.878146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:10.878152 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:16:10.878170 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:10.878171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:10.878183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:10.878273 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:10.878688 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.879144 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.879942 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:10.880244 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:10.880300 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:10.881255 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:10.881256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:10.881256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:10.881263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:10.883085 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.883462 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.883464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.883464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.883542 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:10.883756 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:10.883757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:10.883758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:10.883988 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:10.884076 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:10.884215 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 04:16:10.884315 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:10.914160 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36009
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:10.998713 18457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:10.999022 18457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:10.999065 18457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:11.000214 18457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:11.000289 18457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:11.000324 18457 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 04:16:11.000353 18457 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:11.000394 18457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:11.001874 18457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:11.002147 18457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:11.002386 18457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:11.007642 18464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:11.007661 18463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:11.007686 18466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:11.008124 18457 server_base.cc:1034] running on GCE node
I20260810 04:16:11.008340 18457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:11.008544 18457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:11.009656 18457 hybrid_clock.cc:648] HybridClock initialized: now 1786335371009654 us; error 42 us; skew 500 ppm
Aug 10 04:16:11 dist-test-slave-g350 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:11.016880 18457 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:11.017354 18457 webserver.cc:460] Webserver started at http://127.15.216.67:33683/ using document root <none> and password file <none>
I20260810 04:16:11.017680 18457 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:11.017755 18457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:11.017912 18457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:11.018993 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "8fcbe54795fc48a68dd6a39b886d228e"
format_stamp: "Formatted at 2026-08-10 04:16:11 on dist-test-slave-g350"
server_key: "3c944deb4cead684f2b7aba9d7e354b6"
server_key_iv: "1a7624681c55d9ce2c176926429f4636"
server_key_version: "encryptionkey@0"
I20260810 04:16:11.019403 18457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "8fcbe54795fc48a68dd6a39b886d228e"
format_stamp: "Formatted at 2026-08-10 04:16:11 on dist-test-slave-g350"
server_key: "3c944deb4cead684f2b7aba9d7e354b6"
server_key_iv: "1a7624681c55d9ce2c176926429f4636"
server_key_version: "encryptionkey@0"
I20260810 04:16:11.020793 18457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:11.021579 18473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:11.021802 18457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:11.021924 18457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "8fcbe54795fc48a68dd6a39b886d228e"
format_stamp: "Formatted at 2026-08-10 04:16:11 on dist-test-slave-g350"
server_key: "3c944deb4cead684f2b7aba9d7e354b6"
server_key_iv: "1a7624681c55d9ce2c176926429f4636"
server_key_version: "encryptionkey@0"
I20260810 04:16:11.022011 18457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:11.033883 18457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:11.036180 18457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:11.036355 18457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:11.036862 18457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:11.036923 18457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:11.036966 18457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:11.037006 18457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:11.044361 18457 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:34855
Aug 10 04:16:11 dist-test-slave-g350 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:11.044453 18598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:34855 every 8 connection(s)
I20260810 04:16:11.044759 18457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:11.047963 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.037530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:46199 (local address 127.15.216.126:36009)
0810 04:16:11.037619 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.037621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.037859 (+ 238us) server_negotiation.cc:407] Connection header received
0810 04:16:11.038569 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.038571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.038586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.038655 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:11.039624 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.040477 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.042106 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.042433 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.042525 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.045007 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:11.045008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:11.045009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:11.045020 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:11.046597 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.047002 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.047003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.047003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.047067 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.047319 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.047320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.047321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.047465 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:11.047534 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.047820 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.047900 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:16:11.048098 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.037473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36009 (local address 127.15.216.67:46199)
0810 04:16:11.037722 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.037761 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.038425 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.038737 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.038742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.039161 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:11.039505 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.039512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.040595 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.040596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.042021 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.042025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.042547 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.042548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.042679 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.043266 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:11.043277 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:11.044843 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:11.046704 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:11.046709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:11.046710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:11.046931 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:11.047112 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:11.047113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:11.047113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:11.047196 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:11.047542 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:11.047548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:11.047666 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.047752 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.047876 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":124,"threads_started":1}
I20260810 04:16:11.048699 18599 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36009
I20260810 04:16:11.048776 18599 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:11.048967 18599 heartbeater.cc:511] Master 127.15.216.126:36009 requested a full tablet report, sending...
I20260810 04:16:11.049402 18076 ts_manager.cc:194] Registered new tserver with Master: 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:11.049942 18076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:46199
I20260810 04:16:11.050077 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 18457
I20260810 04:16:11.050163 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:11.050323 16225 external_mini_cluster.cc:1428] Setting key 16be67c166c0fcaed89d8183fdc97e9c
I20260810 04:16:11.051091 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:16:11.056517 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.049881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:41593 (local address 127.15.216.126:36009)
0810 04:16:11.049926 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.049928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.050570 (+ 642us) server_negotiation.cc:407] Connection header received
0810 04:16:11.050645 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.050646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.050656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.050704 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:11.051118 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.051609 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.052414 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.052661 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.052747 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.053904 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:11.053905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:11.053905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:11.053912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:11.055261 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.055669 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.055670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.055670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.055719 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.055963 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.055963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.055963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.056118 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:11.056178 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.056394 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.056458 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:16:11.056696 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.049789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36009 (local address 127.15.216.67:41593)
0810 04:16:11.050513 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.050525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.050583 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.050750 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.050751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.050968 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:11.051033 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.051035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.051709 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.051710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.052329 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.052331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.052764 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.052765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.052845 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.053304 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:11.053309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:11.053759 (+ 450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:11.055349 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:11.055350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:11.055351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:11.055589 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:11.055797 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:11.055798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:11.055798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:11.055876 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:11.056194 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:11.056195 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:11.056255 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.056339 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.056454 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":95,"threads_started":1}
Aug 10 04:16:11 dist-test-slave-g350 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:16:11 dist-test-slave-g350 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:11.076395 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.069061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.15.216.126:36009)
0810 04:16:11.069105 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.069107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.069115 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:11.069142 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.069143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.069154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.069225 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:11.069872 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.070388 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.071161 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.071357 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.071435 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.073731 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:11.073732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:11.073733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:11.073739 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:11.075120 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.075542 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.075543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.075543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.075604 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.075818 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.075819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.075820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.075947 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:11.076055 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.076189 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.076287 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:16:11.078606 18076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:11.080747 18076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:11.087152 18620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.083049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38125 (local address 127.0.0.1:40756)
0810 04:16:11.083275 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.083318 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.083388 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.083654 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.083657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.083659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.083733 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.083735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.084814 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.084816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.085973 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.085976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.086848 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.086849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.086927 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.086942 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.087015 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.087071 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":63,"threads_started":1}
I20260810 04:16:11.087195 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.083171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.15.216.66:38125)
0810 04:16:11.083385 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.083390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.083402 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:16:11.083489 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.083491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.083507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.083579 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.083940 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.084716 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.086062 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.086765 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.086848 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.086913 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.087047 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.087092 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260810 04:16:11.087728 18621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.083122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34855 (local address 127.0.0.1:59894)
0810 04:16:11.083554 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.083566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.083625 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.083994 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.083995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.083996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.084027 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.084028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.085329 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.085330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.086306 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.086309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.087503 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.087504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.087570 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.087573 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.087624 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.087665 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":38,"thread_start_us":99,"threads_started":1}
I20260810 04:16:11.087976 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.083122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33881 (local address 127.0.0.1:57592)
0810 04:16:11.083589 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.083598 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.083652 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.083849 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.083850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.083852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.083920 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.083921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.085338 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.085339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.086687 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.086690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.087729 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.087730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.087808 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.087812 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.087862 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.087907 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":25,"thread_start_us":102,"threads_started":1}
I20260810 04:16:11.087898 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 971da38e13be4d1fa611fdee732a197b (DEFAULT_TABLE table=test-table [id=163264cff5e649bdb0531b57715636c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:11.088270 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.083379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.15.216.67:34855)
0810 04:16:11.083757 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.083760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.083768 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:11.083830 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.083832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.083848 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.083924 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.084104 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.085228 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.086410 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.087195 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.087406 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.087470 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.087655 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.087699 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":82,"threads_started":1}
I20260810 04:16:11.088315 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971da38e13be4d1fa611fdee732a197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.088315 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.083412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.15.216.65:33881)
0810 04:16:11.083586 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.083589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.083644 (+ 55us) server_negotiation.cc:407] Connection header received
0810 04:16:11.083730 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.083732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.083749 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.083840 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.084009 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.085221 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.086797 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.087571 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.087642 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.087690 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.088163 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.088205 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260810 04:16:11.088901 18527 tablet_service.cc:1467] Processing CreateTablet for tablet 971da38e13be4d1fa611fdee732a197b (DEFAULT_TABLE table=test-table [id=163264cff5e649bdb0531b57715636c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:11.089214 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971da38e13be4d1fa611fdee732a197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.090206 18232 tablet_service.cc:1467] Processing CreateTablet for tablet 971da38e13be4d1fa611fdee732a197b (DEFAULT_TABLE table=test-table [id=163264cff5e649bdb0531b57715636c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:11.090544 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971da38e13be4d1fa611fdee732a197b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.092065 18625 tablet_bootstrap.cc:492] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Bootstrap starting.
I20260810 04:16:11.092890 18625 tablet_bootstrap.cc:654] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.093215 18625 log.cc:824] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:11.093891 18626 tablet_bootstrap.cc:492] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Bootstrap starting.
I20260810 04:16:11.094106 18625 tablet_bootstrap.cc:492] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: No bootstrap required, opened a new log
I20260810 04:16:11.094187 18625 ts_tablet_manager.cc:1397] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:11.094266 18628 tablet_bootstrap.cc:492] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Bootstrap starting.
I20260810 04:16:11.094602 18626 tablet_bootstrap.cc:654] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.094888 18626 log.cc:824] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:11.095120 18628 tablet_bootstrap.cc:654] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.095467 18628 log.cc:824] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:11.095652 18626 tablet_bootstrap.cc:492] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: No bootstrap required, opened a new log
I20260810 04:16:11.095752 18626 ts_tablet_manager.cc:1397] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:11.096287 18628 tablet_bootstrap.cc:492] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: No bootstrap required, opened a new log
I20260810 04:16:11.096405 18628 ts_tablet_manager.cc:1397] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:16:11.096726 18625 raft_consensus.cc:348] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.096930 18625 raft_consensus.cc:374] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.096974 18625 raft_consensus.cc:729] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcbe54795fc48a68dd6a39b886d228e, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.097093 18625 consensus_queue.cc:260] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.097414 18599 heartbeater.cc:503] Master 127.15.216.126:36009 was elected leader, sending a full tablet report...
I20260810 04:16:11.097359 18626 raft_consensus.cc:348] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.097545 18626 raft_consensus.cc:374] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.097573 18626 raft_consensus.cc:729] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d16f92878430ca0bd54598124613c, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.097693 18625 ts_tablet_manager.cc:1428] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:11.097731 18626 consensus_queue.cc:260] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.098055 18448 heartbeater.cc:503] Master 127.15.216.126:36009 was elected leader, sending a full tablet report...
I20260810 04:16:11.098055 18626 ts_tablet_manager.cc:1428] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:16:11.099287 18628 raft_consensus.cc:348] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.099404 18628 raft_consensus.cc:374] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.099426 18628 raft_consensus.cc:729] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f5aa45ed7744f19f869ec1a1ad8092, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.099531 18628 consensus_queue.cc:260] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.099800 18628 ts_tablet_manager.cc:1428] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:11.099879 18302 heartbeater.cc:503] Master 127.15.216.126:36009 was elected leader, sending a full tablet report...
W20260810 04:16:11.124342 18449 tablet.cc:2369] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:16:11.205154 18304 tablet.cc:2369] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:11.214414 18632 raft_consensus.cc:482] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:11.214555 18632 raft_consensus.cc:504] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.215052 18632 leader_election.cc:288] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:11.218245 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.215433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:55253 (local address 127.15.216.67:34855)
0810 04:16:11.215489 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.215491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.215500 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:11.215531 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.215532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.215542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.215600 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.215824 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.216362 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.217383 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.217862 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.217929 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.217936 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.218126 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.218171 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:16:11.218343 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.215439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:60233 (local address 127.15.216.65:33881)
0810 04:16:11.215489 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.215491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.215618 (+ 127us) server_negotiation.cc:407] Connection header received
0810 04:16:11.215652 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.215653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.215685 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.215766 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.215982 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.216529 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.217492 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.217951 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.218035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.218043 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.218223 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.218270 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:16:11.218317 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.215267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34855 (local address 127.15.216.66:55253)
0810 04:16:11.215327 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.215339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.215396 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.215648 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.215650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.215653 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.215742 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.215745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.216455 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.216456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.217280 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.217283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.217946 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.217947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.218039 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.218043 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.218112 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.218147 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 04:16:11.218339 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.215334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33881 (local address 127.15.216.66:60233)
0810 04:16:11.215522 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.215533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.215591 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.215774 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.215775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.215776 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.215889 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.215890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.216624 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.216625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.217382 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.217384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.218042 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.218043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.218136 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.218140 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.218199 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.218236 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":121,"threads_started":1}
I20260810 04:16:11.218775 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971da38e13be4d1fa611fdee732a197b" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e" is_pre_election: true
I20260810 04:16:11.218986 18550 raft_consensus.cc:2454] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 0.
I20260810 04:16:11.218966 18253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971da38e13be4d1fa611fdee732a197b" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" is_pre_election: true
I20260810 04:16:11.219130 18253 raft_consensus.cc:2454] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 0.
I20260810 04:16:11.219283 18330 leader_election.cc:302] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 8fcbe54795fc48a68dd6a39b886d228e; no voters:
I20260810 04:16:11.219417 18632 raft_consensus.cc:2781] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:11.219481 18632 raft_consensus.cc:482] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:11.219514 18632 raft_consensus.cc:3037] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.220333 18632 raft_consensus.cc:504] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.220510 18632 leader_election.cc:288] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 election: Requested vote from peers 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:11.220669 18253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971da38e13be4d1fa611fdee732a197b" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5aa45ed7744f19f869ec1a1ad8092"
I20260810 04:16:11.220710 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971da38e13be4d1fa611fdee732a197b" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e"
I20260810 04:16:11.220758 18253 raft_consensus.cc:3037] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.220820 18550 raft_consensus.cc:3037] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.221525 18253 raft_consensus.cc:2454] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 1.
I20260810 04:16:11.221671 18550 raft_consensus.cc:2454] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 1.
I20260810 04:16:11.222071 18331 leader_election.cc:302] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 97f5aa45ed7744f19f869ec1a1ad8092; no voters:
I20260810 04:16:11.222203 18632 raft_consensus.cc:2781] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:11.222376 18632 raft_consensus.cc:686] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 1 LEADER]: Becoming Leader. State: Replica: 6a7d16f92878430ca0bd54598124613c, State: Running, Role: LEADER
I20260810 04:16:11.222492 18632 consensus_queue.cc:237] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.223587 18075 catalog_manager.cc:5495] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7d16f92878430ca0bd54598124613c (127.15.216.66). New cstate: current_term: 1 leader_uuid: "6a7d16f92878430ca0bd54598124613c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } health_report { overall_health: UNKNOWN } } }
Aug 10 04:16:11 dist-test-slave-g350 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335370, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:11.253711 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.246362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36009 (local address 127.0.0.1:36416)
0810 04:16:11.246587 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.246603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.246684 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.246836 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.246837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.247150 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:11.247195 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.247197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.247929 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.247930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.248504 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.248506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.248934 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.248935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.248992 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.249556 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:11.249561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:11.251039 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:11.252581 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:11.252589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:11.252592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:11.252793 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:11.253030 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:11.253030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:11.253031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:11.253091 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:11.253352 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:11.253359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:11.253408 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.253465 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.253562 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":115,"threads_started":1}
I20260810 04:16:11.253773 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.246471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.15.216.126:36009)
0810 04:16:11.246557 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.246559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.246665 (+ 106us) server_negotiation.cc:407] Connection header received
0810 04:16:11.246756 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.246757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.246768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.246825 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:11.247319 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.247842 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.248597 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.248828 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.248918 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.251124 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:11.251125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:11.251126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:11.251132 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:11.252450 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.252914 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.252915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.252915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.252968 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:11.253146 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:11.253147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:11.253147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:11.253249 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:11.253348 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.253502 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.253680 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":53}
I20260810 04:16:11.255685 18075 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:36416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:16:11.255867 18075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:11.257395 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 72f6f59537af487ca527288c3244fa5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6f3f7c0339c49709a5d47b0c31c3fed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:11.257386 18232 tablet_service.cc:1467] Processing CreateTablet for tablet 72f6f59537af487ca527288c3244fa5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6f3f7c0339c49709a5d47b0c31c3fed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:11.257431 18527 tablet_service.cc:1467] Processing CreateTablet for tablet 72f6f59537af487ca527288c3244fa5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c6f3f7c0339c49709a5d47b0c31c3fed]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:11.257561 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f6f59537af487ca527288c3244fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.257572 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f6f59537af487ca527288c3244fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.257624 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f6f59537af487ca527288c3244fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.259187 18628 tablet_bootstrap.cc:492] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: Bootstrap starting.
I20260810 04:16:11.259191 18625 tablet_bootstrap.cc:492] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: Bootstrap starting.
I20260810 04:16:11.259573 18628 tablet_bootstrap.cc:654] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.259819 18625 tablet_bootstrap.cc:654] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.260224 18626 tablet_bootstrap.cc:492] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: Bootstrap starting.
I20260810 04:16:11.260716 18626 tablet_bootstrap.cc:654] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.261333 18628 tablet_bootstrap.cc:492] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: No bootstrap required, opened a new log
I20260810 04:16:11.261376 18628 ts_tablet_manager.cc:1397] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:11.261404 18625 tablet_bootstrap.cc:492] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: No bootstrap required, opened a new log
I20260810 04:16:11.261451 18625 ts_tablet_manager.cc:1397] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:11.261551 18628 raft_consensus.cc:348] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.261677 18628 raft_consensus.cc:374] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.261713 18628 raft_consensus.cc:729] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f5aa45ed7744f19f869ec1a1ad8092, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.261649 18625 raft_consensus.cc:348] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.261742 18625 raft_consensus.cc:374] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.261770 18625 raft_consensus.cc:729] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcbe54795fc48a68dd6a39b886d228e, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.261761 18628 consensus_queue.cc:260] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.261827 18626 tablet_bootstrap.cc:492] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: No bootstrap required, opened a new log
I20260810 04:16:11.261827 18625 consensus_queue.cc:260] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.261863 18628 ts_tablet_manager.cc:1428] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:11.261878 18626 ts_tablet_manager.cc:1397] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:11.261929 18625 ts_tablet_manager.cc:1428] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:11.262004 18633 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.262085 18633 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:11.262030 18626 raft_consensus.cc:348] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.262130 18626 raft_consensus.cc:374] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.262156 18626 raft_consensus.cc:729] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d16f92878430ca0bd54598124613c, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.262207 18626 consensus_queue.cc:260] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.262234 18631 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.262307 18626 ts_tablet_manager.cc:1428] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:11.262331 18631 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:11.262539 18632 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.262629 18632 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 04:16:11.295454 18600 tablet.cc:2369] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:11.362578 18631 raft_consensus.cc:482] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:11.362708 18631 raft_consensus.cc:504] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.363123 18631 leader_election.cc:288] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881), 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125)
I20260810 04:16:11.366293 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.363314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38125 (local address 127.15.216.67:44295)
0810 04:16:11.363384 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.363396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.363444 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.363606 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.363607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.363609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.363709 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.363711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.364566 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.364567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.365397 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.365400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.366039 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.366040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.366135 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.366140 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.366195 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.366237 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":19}
I20260810 04:16:11.366385 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.363407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:44295 (local address 127.15.216.66:38125)
0810 04:16:11.363455 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.363457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.363469 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:16:11.363503 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.363504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.363513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.363595 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.363810 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.364477 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.365486 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.365948 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.366010 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.366019 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.366214 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.366306 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:16:11.366484 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.363314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33881 (local address 127.15.216.67:50123)
0810 04:16:11.363576 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.363587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.363650 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.363805 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.363806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.363808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.363881 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.363883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.364591 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.364597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.365412 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.365415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.366213 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.366214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.366285 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.366290 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.366353 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.366390 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":40,"thread_start_us":161,"threads_started":1}
I20260810 04:16:11.366503 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.363408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:50123 (local address 127.15.216.65:33881)
0810 04:16:11.363457 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.363459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.363664 (+ 205us) server_negotiation.cc:407] Connection header received
0810 04:16:11.363702 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.363703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.363713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.363774 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.363953 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.364497 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.365561 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.366091 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.366166 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.366176 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.366380 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.366434 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:16:11.366647 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6f59537af487ca527288c3244fa5d" candidate_uuid: "8fcbe54795fc48a68dd6a39b886d228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7d16f92878430ca0bd54598124613c" is_pre_election: true
I20260810 04:16:11.366762 18400 raft_consensus.cc:2454] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fcbe54795fc48a68dd6a39b886d228e in term 0.
I20260810 04:16:11.366730 18253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6f59537af487ca527288c3244fa5d" candidate_uuid: "8fcbe54795fc48a68dd6a39b886d228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" is_pre_election: true
I20260810 04:16:11.366801 18253 raft_consensus.cc:2454] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fcbe54795fc48a68dd6a39b886d228e in term 0.
I20260810 04:16:11.366998 18475 leader_election.cc:302] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 8fcbe54795fc48a68dd6a39b886d228e; no voters:
I20260810 04:16:11.367177 18631 raft_consensus.cc:2781] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:11.367211 18631 raft_consensus.cc:482] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:11.367226 18631 raft_consensus.cc:3037] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.367956 18631 raft_consensus.cc:504] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.368104 18631 leader_election.cc:288] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [CANDIDATE]: Term 1 election: Requested vote from peers 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881), 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125)
I20260810 04:16:11.368257 18253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6f59537af487ca527288c3244fa5d" candidate_uuid: "8fcbe54795fc48a68dd6a39b886d228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5aa45ed7744f19f869ec1a1ad8092"
I20260810 04:16:11.368342 18253 raft_consensus.cc:3037] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.368335 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6f59537af487ca527288c3244fa5d" candidate_uuid: "8fcbe54795fc48a68dd6a39b886d228e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7d16f92878430ca0bd54598124613c"
I20260810 04:16:11.368422 18400 raft_consensus.cc:3037] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.368948 18253 raft_consensus.cc:2454] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fcbe54795fc48a68dd6a39b886d228e in term 1.
I20260810 04:16:11.369108 18400 raft_consensus.cc:2454] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fcbe54795fc48a68dd6a39b886d228e in term 1.
I20260810 04:16:11.369268 18475 leader_election.cc:302] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 8fcbe54795fc48a68dd6a39b886d228e; no voters:
I20260810 04:16:11.369370 18631 raft_consensus.cc:2781] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:11.369511 18631 raft_consensus.cc:686] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [term 1 LEADER]: Becoming Leader. State: Replica: 8fcbe54795fc48a68dd6a39b886d228e, State: Running, Role: LEADER
I20260810 04:16:11.369637 18631 consensus_queue.cc:237] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.370146 18648 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: TxnStatusTablet state changed. Reason: New leader 8fcbe54795fc48a68dd6a39b886d228e. Latest consensus state: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.370275 18648 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:11.370548 18650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:16:11.370486 18076 catalog_manager.cc:5495] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67). New cstate: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } health_report { overall_health: HEALTHY } } }
I20260810 04:16:11.370666 18650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:16:11.380411 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.377487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34855 (local address 127.0.0.1:59896)
0810 04:16:11.377582 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.377610 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.377665 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.377849 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.377849 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.377850 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:11.377902 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.377904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.378804 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.378805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.379360 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.379362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.379850 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.379851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.379911 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.379929 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.380245 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.380256 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.380301 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.380345 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:16:11.380409 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.377618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.15.216.67:34855)
0810 04:16:11.377663 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.377665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.377675 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:11.377748 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.377749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.377769 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.377845 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:11.377966 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.378723 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.379446 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.379735 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.379811 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.380034 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.380154 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.380246 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.380305 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.380327 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:16:11.382244 18400 raft_consensus.cc:1261] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Refusing update from remote peer 8fcbe54795fc48a68dd6a39b886d228e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:11.382542 18253 raft_consensus.cc:1261] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Refusing update from remote peer 8fcbe54795fc48a68dd6a39b886d228e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:11.382885 18631 consensus_queue.cc:1035] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:11.383040 18648 consensus_queue.cc:1035] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:11.384287 18633 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: TxnStatusTablet state changed. Reason: New leader 8fcbe54795fc48a68dd6a39b886d228e. Latest consensus state: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.384375 18633 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:11.385475 18632 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: TxnStatusTablet state changed. Reason: New leader 8fcbe54795fc48a68dd6a39b886d228e. Latest consensus state: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.385565 18632 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:11.386359 18648 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.386463 18648 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:11.386515 18648 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.386554 18648 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 8fcbe54795fc48a68dd6a39b886d228e: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:11.386672 18633 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.386749 18633 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 97f5aa45ed7744f19f869ec1a1ad8092: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:11.386734 18632 tablet_replica.cc:439] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fcbe54795fc48a68dd6a39b886d228e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } } }
I20260810 04:16:11.386811 18632 tablet_replica.cc:442] T 72f6f59537af487ca527288c3244fa5d P 6a7d16f92878430ca0bd54598124613c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:11.386945 18657 mvcc.cc:205] Tried to move back new op lower bound from 7316829681178828800 to 7316829681130553344. Current Snapshot: MvccSnapshot[applied={T|T < 7316829681178828800}]
I20260810 04:16:11.387157 18655 mvcc.cc:205] Tried to move back new op lower bound from 7316829681178828800 to 7316829681130553344. Current Snapshot: MvccSnapshot[applied={T|T < 7316829681178828800}]
I20260810 04:16:11.391681 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.389226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.15.216.66:38125)
0810 04:16:11.389263 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.389265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.389290 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:16:11.389376 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.389376 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.389392 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.389467 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:11.389660 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.390117 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.390884 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.391097 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.391185 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.391371 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.391487 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.391572 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.391594 (+ 22us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.391613 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:16:11.399986 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.394120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:50161 (local address 127.15.216.67:34855)
0810 04:16:11.394157 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.394165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.397227 (+ 3062us) server_negotiation.cc:407] Connection header received
0810 04:16:11.397292 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.397293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.397302 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.397349 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:11.397496 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.398061 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.399273 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.399479 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.399534 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.399764 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.399790 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.399825 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.399913 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.399930 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:16:11.400049 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.394063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34855 (local address 127.15.216.66:50161)
0810 04:16:11.397146 (+ 3083us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.397158 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.397234 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.397351 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.397352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.397353 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:11.397411 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.397412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.398169 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.398170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.399185 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.399188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.399563 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.399564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.399637 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.399654 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.399853 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.399856 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.399911 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.399956 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3041,"thread_start_us":82,"threads_started":1}
I20260810 04:16:11.402864 18253 raft_consensus.cc:1261] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Refusing update from remote peer 6a7d16f92878430ca0bd54598124613c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:11.402880 18550 raft_consensus.cc:1261] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Refusing update from remote peer 6a7d16f92878430ca0bd54598124613c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:11.403111 18635 consensus_queue.cc:1035] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:11.403293 18632 consensus_queue.cc:1035] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:11.412293 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.409248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38125 (local address 127.15.216.67:47553)
0810 04:16:11.409313 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.409325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.409373 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.409646 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.409646 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.409648 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:11.409716 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.409718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.410606 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.410607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.411478 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.411481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.411930 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.411931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.411982 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.411997 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.412168 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.412170 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.412205 (+ 35us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.412238 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260810 04:16:11.412308 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.409447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:47553 (local address 127.15.216.66:38125)
0810 04:16:11.409515 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.409517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.409526 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:11.409548 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.409549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.409560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.409637 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:11.409788 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.410526 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.411540 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.411841 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.411910 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.412078 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.412111 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.412163 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.412215 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.412235 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:16:11.421911 18075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:11.422040 18075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:11.422777 18075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Sending DeleteTablet for 3 replicas of tablet 971da38e13be4d1fa611fdee732a197b
I20260810 04:16:11.423103 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet 971da38e13be4d1fa611fdee732a197b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:11 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:40756
I20260810 04:16:11.423230 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet 971da38e13be4d1fa611fdee732a197b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:11 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:57592
I20260810 04:16:11.423499 18527 tablet_service.cc:1514] Processing DeleteTablet for tablet 971da38e13be4d1fa611fdee732a197b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:11 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:59894
I20260810 04:16:11.423761 18677 tablet_replica.cc:330] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: stopping tablet replica
I20260810 04:16:11.423732 18075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:11.423893 18075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:11.424201 18677 raft_consensus.cc:2229] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:11.424345 18677 raft_consensus.cc:2258] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:11.424934 18677 ts_tablet_manager.cc:1905] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:11.425325 18379 tablet_service.cc:1467] Processing CreateTablet for tablet ea9c2df292e648a1889d13a9a98bb905 (DEFAULT_TABLE table=test-table [id=1c7ebf038d1947738247c90e03287310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:11.425484 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9c2df292e648a1889d13a9a98bb905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.426927 18626 tablet_bootstrap.cc:492] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: Bootstrap starting.
I20260810 04:16:11.427336 18626 tablet_bootstrap.cc:654] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.428059 18232 tablet_service.cc:1467] Processing CreateTablet for tablet ea9c2df292e648a1889d13a9a98bb905 (DEFAULT_TABLE table=test-table [id=1c7ebf038d1947738247c90e03287310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:11.428208 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9c2df292e648a1889d13a9a98bb905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.429185 18628 tablet_bootstrap.cc:492] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: Bootstrap starting.
I20260810 04:16:11.429445 18527 tablet_service.cc:1467] Processing CreateTablet for tablet ea9c2df292e648a1889d13a9a98bb905 (DEFAULT_TABLE table=test-table [id=1c7ebf038d1947738247c90e03287310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:11.429746 18626 tablet_bootstrap.cc:492] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: No bootstrap required, opened a new log
I20260810 04:16:11.429792 18626 ts_tablet_manager.cc:1397] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 04:16:11.429961 18677 ts_tablet_manager.cc:1918] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:11.429987 18628 tablet_bootstrap.cc:654] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.429952 18626 raft_consensus.cc:348] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.430027 18626 raft_consensus.cc:374] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.430033 18677 log.cc:1196] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/971da38e13be4d1fa611fdee732a197b
I20260810 04:16:11.430050 18626 raft_consensus.cc:729] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d16f92878430ca0bd54598124613c, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.430107 18626 consensus_queue.cc:260] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.430217 18626 ts_tablet_manager.cc:1428] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 04:16:11.430331 18677 ts_tablet_manager.cc:1939] T 971da38e13be4d1fa611fdee732a197b P 8fcbe54795fc48a68dd6a39b886d228e: Deleting consensus metadata
I20260810 04:16:11.430521 18675 tablet_replica.cc:330] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: stopping tablet replica
I20260810 04:16:11.430629 18675 raft_consensus.cc:2229] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:11.430724 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9c2df292e648a1889d13a9a98bb905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:11.430735 18675 raft_consensus.cc:2258] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:11.431447 18671 raft_consensus.cc:482] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:11.431499 18671 raft_consensus.cc:504] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.431628 18671 leader_election.cc:288] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:11.431826 18253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c2df292e648a1889d13a9a98bb905" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" is_pre_election: true
I20260810 04:16:11.431979 18625 tablet_bootstrap.cc:492] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: Bootstrap starting.
I20260810 04:16:11.432080 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c2df292e648a1889d13a9a98bb905" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e" is_pre_election: true
W20260810 04:16:11.432114 18331 leader_election.cc:341] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881): Illegal state: must be running to vote when last-logged opid is not known
W20260810 04:16:11.432400 18330 leader_election.cc:341] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855): Illegal state: must be running to vote when last-logged opid is not known
I20260810 04:16:11.432456 18330 leader_election.cc:302] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c; no voters: 8fcbe54795fc48a68dd6a39b886d228e, 97f5aa45ed7744f19f869ec1a1ad8092
I20260810 04:16:11.432461 18625 tablet_bootstrap.cc:654] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:11.432564 18671 raft_consensus.cc:2726] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 04:16:11.432775 18675 ts_tablet_manager.cc:1905] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:11.432947 18058 catalog_manager.cc:4841] TS 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855): tablet 971da38e13be4d1fa611fdee732a197b (table test-table [id=163264cff5e649bdb0531b57715636c7]) successfully deleted
I20260810 04:16:11.433192 18625 tablet_bootstrap.cc:492] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: No bootstrap required, opened a new log
I20260810 04:16:11.433261 18625 ts_tablet_manager.cc:1397] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:11.433419 18625 raft_consensus.cc:348] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.433493 18625 raft_consensus.cc:374] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.433522 18625 raft_consensus.cc:729] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcbe54795fc48a68dd6a39b886d228e, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.433568 18625 consensus_queue.cc:260] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.433697 18625 ts_tablet_manager.cc:1428] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:11.433849 18676 tablet_replica.cc:330] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: stopping tablet replica
I20260810 04:16:11.434019 18676 raft_consensus.cc:2229] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:11.434139 18676 raft_consensus.cc:2258] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:11.434438 18675 ts_tablet_manager.cc:1918] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:11.434517 18675 log.cc:1196] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/971da38e13be4d1fa611fdee732a197b
I20260810 04:16:11.434633 18676 ts_tablet_manager.cc:1905] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:11.434708 18628 tablet_bootstrap.cc:492] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: No bootstrap required, opened a new log
I20260810 04:16:11.434742 18628 ts_tablet_manager.cc:1397] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 04:16:11.434810 18675 ts_tablet_manager.cc:1939] T 971da38e13be4d1fa611fdee732a197b P 6a7d16f92878430ca0bd54598124613c: Deleting consensus metadata
I20260810 04:16:11.434880 18628 raft_consensus.cc:348] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.434938 18628 raft_consensus.cc:374] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:11.434957 18628 raft_consensus.cc:729] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f5aa45ed7744f19f869ec1a1ad8092, State: Initialized, Role: FOLLOWER
I20260810 04:16:11.434999 18628 consensus_queue.cc:260] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.435084 18628 ts_tablet_manager.cc:1428] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:11.435179 18057 catalog_manager.cc:4841] TS 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125): tablet 971da38e13be4d1fa611fdee732a197b (table test-table [id=163264cff5e649bdb0531b57715636c7]) successfully deleted
I20260810 04:16:11.436362 18676 ts_tablet_manager.cc:1918] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:11.436467 18676 log.cc:1196] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/971da38e13be4d1fa611fdee732a197b
I20260810 04:16:11.436751 18676 ts_tablet_manager.cc:1939] T 971da38e13be4d1fa611fdee732a197b P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting consensus metadata
I20260810 04:16:11.437158 18059 catalog_manager.cc:4841] TS 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881): tablet 971da38e13be4d1fa611fdee732a197b (table test-table [id=163264cff5e649bdb0531b57715636c7]) successfully deleted
I20260810 04:16:11.647781 18633 raft_consensus.cc:482] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:11.647876 18633 raft_consensus.cc:504] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.648278 18633 leader_election.cc:288] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:11.651465 18622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.648572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:34781 (local address 127.15.216.66:38125)
0810 04:16:11.648628 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.648630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.648747 (+ 117us) server_negotiation.cc:407] Connection header received
0810 04:16:11.648826 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.648828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.648844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.648902 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.649100 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.649693 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.650750 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.651112 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.651169 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.651176 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.651349 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.651394 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:16:11.651480 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.648490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38125 (local address 127.15.216.65:34781)
0810 04:16:11.648624 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.648666 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.648757 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.648921 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.648922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.648925 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.649015 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.649017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.649775 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.649776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.650647 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.650649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.651197 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.651198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.651272 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.651276 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.651333 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.651364 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":28,"thread_start_us":68,"threads_started":1}
I20260810 04:16:11.651541 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.648678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:51857 (local address 127.15.216.67:34855)
0810 04:16:11.648752 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.648755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.648789 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:16:11.648872 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.648873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.648917 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.648978 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:11.649191 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.649742 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.650762 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.651173 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.651243 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.651251 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.651427 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.651465 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:16:11.651749 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.648573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34855 (local address 127.15.216.65:51857)
0810 04:16:11.648726 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.648737 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.648807 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.648995 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.648996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.649011 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:11.649117 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.649119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.649827 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.649828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.650648 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.650651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.651261 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.651262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.651343 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.651348 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.651406 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.651444 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":18,"thread_start_us":56,"threads_started":1}
I20260810 04:16:11.651981 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c2df292e648a1889d13a9a98bb905" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7d16f92878430ca0bd54598124613c" is_pre_election: true
I20260810 04:16:11.652094 18400 raft_consensus.cc:2454] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 0.
I20260810 04:16:11.652076 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c2df292e648a1889d13a9a98bb905" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e" is_pre_election: true
I20260810 04:16:11.652189 18550 raft_consensus.cc:2454] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 0.
I20260810 04:16:11.652315 18178 leader_election.cc:302] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 97f5aa45ed7744f19f869ec1a1ad8092; no voters:
I20260810 04:16:11.652454 18633 raft_consensus.cc:2781] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:11.652505 18633 raft_consensus.cc:482] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:11.652530 18633 raft_consensus.cc:3037] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.653256 18633 raft_consensus.cc:504] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.653429 18633 leader_election.cc:288] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 election: Requested vote from peers 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:11.653635 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c2df292e648a1889d13a9a98bb905" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7d16f92878430ca0bd54598124613c"
I20260810 04:16:11.653723 18400 raft_consensus.cc:3037] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.653728 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9c2df292e648a1889d13a9a98bb905" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e"
I20260810 04:16:11.653795 18550 raft_consensus.cc:3037] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:11.654562 18550 raft_consensus.cc:2454] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 1.
I20260810 04:16:11.654562 18400 raft_consensus.cc:2454] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 1.
I20260810 04:16:11.654724 18178 leader_election.cc:302] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 97f5aa45ed7744f19f869ec1a1ad8092; no voters:
I20260810 04:16:11.654804 18633 raft_consensus.cc:2781] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:11.654960 18633 raft_consensus.cc:686] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 LEADER]: Becoming Leader. State: Replica: 97f5aa45ed7744f19f869ec1a1ad8092, State: Running, Role: LEADER
I20260810 04:16:11.655049 18633 consensus_queue.cc:237] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:11.655886 18075 catalog_manager.cc:5495] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65). New cstate: current_term: 1 leader_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } health_report { overall_health: UNKNOWN } } }
I20260810 04:16:11.665977 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.663453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.15.216.65:33881)
0810 04:16:11.663522 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.663524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.663533 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:11.663619 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.663620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.663642 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.663732 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:11.663843 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.664346 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.665135 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.665344 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.665422 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.665579 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.665716 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.665835 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.665878 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.665901 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:16:11.670945 18623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.668412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:38699 (local address 127.15.216.67:34855)
0810 04:16:11.668470 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.668472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.668583 (+ 111us) server_negotiation.cc:407] Connection header received
0810 04:16:11.668650 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.668650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.668660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.668715 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:11.668865 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.669311 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.670211 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.670423 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.670508 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.670679 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.670712 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.670779 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.670842 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.670860 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:16:11.670979 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.668321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34855 (local address 127.15.216.65:38699)
0810 04:16:11.668507 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.668518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.668599 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.668719 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.668719 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.668720 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:11.668794 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.668795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.669403 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.669404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.670125 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.670127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.670513 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.670514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.670578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.670597 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.670780 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.670783 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.670837 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.670877 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260810 04:16:11.673309 18550 raft_consensus.cc:1261] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Refusing update from remote peer 97f5aa45ed7744f19f869ec1a1ad8092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:11.673341 18400 raft_consensus.cc:1261] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Refusing update from remote peer 97f5aa45ed7744f19f869ec1a1ad8092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:11.673874 18684 consensus_queue.cc:1035] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:11.673998 18633 consensus_queue.cc:1035] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:11.675783 18654 mvcc.cc:205] Tried to move back new op lower bound from 7316829682372050944 to 7316829682299580416. Current Snapshot: MvccSnapshot[applied={T|T < 7316829682372050944}]
I20260810 04:16:11.681625 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.679421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:37601 (local address 127.15.216.65:33881)
0810 04:16:11.679462 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:11.679464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:11.679474 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:11.679522 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:11.679522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:11.679533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:11.679602 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:11.679677 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.680168 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.680888 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.681134 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.681217 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.681373 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.681410 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.681458 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:11.681517 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 04:16:11.681536 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:16:11.681661 18606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:11.679360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33881 (local address 127.15.216.67:37601)
0810 04:16:11.679417 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:11.679428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:11.679465 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:11.679603 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:11.679603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:11.679604 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:11.679639 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.679640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.680233 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.680233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.680799 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:11.680802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:11.681217 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:11.681218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:11.681276 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:11.681280 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:11.681471 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:11.681473 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:11.681517 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:16:11.681560 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:16:11.681939 18655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 04:16:13.202584 18073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:13.202718 18073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:13.203727 18073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Sending DeleteTablet for 3 replicas of tablet ea9c2df292e648a1889d13a9a98bb905
I20260810 04:16:13.204064 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9c2df292e648a1889d13a9a98bb905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:13 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:40756
I20260810 04:16:13.204124 18527 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9c2df292e648a1889d13a9a98bb905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:13 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:59894
I20260810 04:16:13.204437 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9c2df292e648a1889d13a9a98bb905 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:13 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:57592
I20260810 04:16:13.204605 18717 tablet_replica.cc:330] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: stopping tablet replica
I20260810 04:16:13.204663 18717 raft_consensus.cc:2229] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:13.204770 18717 raft_consensus.cc:2258] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:13.205030 18717 ts_tablet_manager.cc:1905] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:13.207139 18717 ts_tablet_manager.cc:1918] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:16:13.207226 18717 log.cc:1196] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/ea9c2df292e648a1889d13a9a98bb905
I20260810 04:16:13.207535 18717 ts_tablet_manager.cc:1939] T ea9c2df292e648a1889d13a9a98bb905 P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting consensus metadata
I20260810 04:16:13.207651 18716 tablet_replica.cc:330] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: stopping tablet replica
I20260810 04:16:13.207710 18716 raft_consensus.cc:2229] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:13.207772 18716 raft_consensus.cc:2258] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:13.207898 18715 tablet_replica.cc:330] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: stopping tablet replica
I20260810 04:16:13.207975 18715 raft_consensus.cc:2229] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:13.208020 18715 raft_consensus.cc:2258] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:13.208056 18716 ts_tablet_manager.cc:1905] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:13.210178 18716 ts_tablet_manager.cc:1918] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:16:13.210259 18716 log.cc:1196] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/ea9c2df292e648a1889d13a9a98bb905
I20260810 04:16:13.210501 18716 ts_tablet_manager.cc:1939] T ea9c2df292e648a1889d13a9a98bb905 P 8fcbe54795fc48a68dd6a39b886d228e: Deleting consensus metadata
I20260810 04:16:13.210837 18059 catalog_manager.cc:4841] TS 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881): tablet ea9c2df292e648a1889d13a9a98bb905 (table test-table [id=1c7ebf038d1947738247c90e03287310]) successfully deleted
I20260810 04:16:13.211380 18058 catalog_manager.cc:4841] TS 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855): tablet ea9c2df292e648a1889d13a9a98bb905 (table test-table [id=1c7ebf038d1947738247c90e03287310]) successfully deleted
I20260810 04:16:13.212868 18715 ts_tablet_manager.cc:1905] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:13.214641 18715 ts_tablet_manager.cc:1918] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:16:13.214723 18715 log.cc:1196] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/ea9c2df292e648a1889d13a9a98bb905
I20260810 04:16:13.214984 18715 ts_tablet_manager.cc:1939] T ea9c2df292e648a1889d13a9a98bb905 P 6a7d16f92878430ca0bd54598124613c: Deleting consensus metadata
I20260810 04:16:13.215386 18057 catalog_manager.cc:4841] TS 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125): tablet ea9c2df292e648a1889d13a9a98bb905 (table test-table [id=1c7ebf038d1947738247c90e03287310]) successfully deleted
Aug 10 04:16:13 dist-test-slave-g350 krb5kdc[18024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:16:13 dist-test-slave-g350 krb5kdc[18024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:13.234171 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:13.226175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.15.216.126:36009)
0810 04:16:13.226353 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:13.226355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:13.226410 (+ 55us) server_negotiation.cc:407] Connection header received
0810 04:16:13.226457 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:13.226458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:13.226470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:13.226545 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:13.227108 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:13.227694 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:13.228463 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:13.228730 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:13.228792 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:13.231483 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:13.231485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:13.231485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:13.231492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:13.233077 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:13.233457 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:13.233458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:13.233459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:13.233517 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:13.233751 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:13.233752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:13.233753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:13.233867 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:13.233948 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:13.234034 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:16:13.234099 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260810 04:16:13.235863 18073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:13.236078 18073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:13.237751 18527 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba7137d8e134ad7be04d3672296b3fe (DEFAULT_TABLE table=test-table [id=3faba631e7204e97a7f438f62fa311a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:13.237761 18232 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba7137d8e134ad7be04d3672296b3fe (DEFAULT_TABLE table=test-table [id=3faba631e7204e97a7f438f62fa311a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:13.237922 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba7137d8e134ad7be04d3672296b3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:13.237953 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba7137d8e134ad7be04d3672296b3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:13.239416 18727 tablet_bootstrap.cc:492] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: Bootstrap starting.
I20260810 04:16:13.239472 18728 tablet_bootstrap.cc:492] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: Bootstrap starting.
I20260810 04:16:13.239519 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba7137d8e134ad7be04d3672296b3fe (DEFAULT_TABLE table=test-table [id=3faba631e7204e97a7f438f62fa311a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:13.239657 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba7137d8e134ad7be04d3672296b3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:13.239852 18727 tablet_bootstrap.cc:654] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:13.239867 18728 tablet_bootstrap.cc:654] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:13.240557 18727 tablet_bootstrap.cc:492] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: No bootstrap required, opened a new log
I20260810 04:16:13.240620 18727 ts_tablet_manager.cc:1397] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:13.240707 18728 tablet_bootstrap.cc:492] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: No bootstrap required, opened a new log
I20260810 04:16:13.240762 18728 ts_tablet_manager.cc:1397] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:13.240729 18727 raft_consensus.cc:348] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.240906 18728 raft_consensus.cc:348] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.240993 18728 raft_consensus.cc:374] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:13.241017 18728 raft_consensus.cc:729] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f5aa45ed7744f19f869ec1a1ad8092, State: Initialized, Role: FOLLOWER
I20260810 04:16:13.241060 18728 consensus_queue.cc:260] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.241236 18728 ts_tablet_manager.cc:1428] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:13.241127 18727 raft_consensus.cc:374] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:13.241284 18727 raft_consensus.cc:729] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcbe54795fc48a68dd6a39b886d228e, State: Initialized, Role: FOLLOWER
I20260810 04:16:13.241348 18732 tablet_bootstrap.cc:492] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: Bootstrap starting.
I20260810 04:16:13.241372 18727 consensus_queue.cc:260] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.241689 18727 ts_tablet_manager.cc:1428] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:13.241734 18732 tablet_bootstrap.cc:654] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:13.242344 18732 tablet_bootstrap.cc:492] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: No bootstrap required, opened a new log
I20260810 04:16:13.242398 18732 ts_tablet_manager.cc:1397] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:13.242494 18732 raft_consensus.cc:348] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.242544 18732 raft_consensus.cc:374] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:13.242558 18732 raft_consensus.cc:729] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d16f92878430ca0bd54598124613c, State: Initialized, Role: FOLLOWER
I20260810 04:16:13.242615 18732 consensus_queue.cc:260] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.242777 18732 ts_tablet_manager.cc:1428] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:13.461242 18706 raft_consensus.cc:482] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:13.461365 18706 raft_consensus.cc:504] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.461647 18706 leader_election.cc:288] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:13.461872 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7137d8e134ad7be04d3672296b3fe" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7d16f92878430ca0bd54598124613c" is_pre_election: true
I20260810 04:16:13.461913 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7137d8e134ad7be04d3672296b3fe" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e" is_pre_election: true
I20260810 04:16:13.461997 18400 raft_consensus.cc:2454] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 0.
I20260810 04:16:13.462003 18550 raft_consensus.cc:2454] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 0.
I20260810 04:16:13.462174 18178 leader_election.cc:302] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 97f5aa45ed7744f19f869ec1a1ad8092; no voters:
I20260810 04:16:13.462322 18706 raft_consensus.cc:2781] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:13.462391 18706 raft_consensus.cc:482] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:13.462424 18706 raft_consensus.cc:3037] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:13.463217 18706 raft_consensus.cc:504] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.463378 18706 leader_election.cc:288] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 election: Requested vote from peers 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:13.463578 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7137d8e134ad7be04d3672296b3fe" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a7d16f92878430ca0bd54598124613c"
I20260810 04:16:13.463601 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7137d8e134ad7be04d3672296b3fe" candidate_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e"
I20260810 04:16:13.463667 18400 raft_consensus.cc:3037] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:13.463673 18550 raft_consensus.cc:3037] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:13.464488 18550 raft_consensus.cc:2454] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 1.
I20260810 04:16:13.464488 18400 raft_consensus.cc:2454] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97f5aa45ed7744f19f869ec1a1ad8092 in term 1.
I20260810 04:16:13.464659 18178 leader_election.cc:302] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 97f5aa45ed7744f19f869ec1a1ad8092; no voters:
I20260810 04:16:13.464771 18706 raft_consensus.cc:2781] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:13.464828 18706 raft_consensus.cc:686] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 LEADER]: Becoming Leader. State: Replica: 97f5aa45ed7744f19f869ec1a1ad8092, State: Running, Role: LEADER
I20260810 04:16:13.464879 18706 consensus_queue.cc:237] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.465664 18075 catalog_manager.cc:5495] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65). New cstate: current_term: 1 leader_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } health_report { overall_health: UNKNOWN } } }
I20260810 04:16:13.527915 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:13.525402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.15.216.65:33881)
0810 04:16:13.525566 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:13.525568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:13.525577 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:13.525644 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:13.525644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:13.525657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:13.525754 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:13.525892 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:13.526400 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:13.527163 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:13.527398 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:13.527478 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:13.527635 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:13.527674 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:13.527734 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:13.527786 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 04:16:13.527811 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":115,"threads_started":1}
I20260810 04:16:13.530238 18550 raft_consensus.cc:1261] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Refusing update from remote peer 97f5aa45ed7744f19f869ec1a1ad8092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:13.530263 18400 raft_consensus.cc:1261] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Refusing update from remote peer 97f5aa45ed7744f19f869ec1a1ad8092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:13.530467 18706 consensus_queue.cc:1035] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:13.530603 18684 consensus_queue.cc:1035] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:13.540189 18075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:13.540283 18075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:13.541121 18075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Sending DeleteTablet for 3 replicas of tablet 4ba7137d8e134ad7be04d3672296b3fe
I20260810 04:16:13.541443 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ba7137d8e134ad7be04d3672296b3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:13 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:57592
I20260810 04:16:13.541501 18527 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ba7137d8e134ad7be04d3672296b3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:13 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:59894
I20260810 04:16:13.541558 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ba7137d8e134ad7be04d3672296b3fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:13 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:40756
I20260810 04:16:13.541651 18716 tablet_replica.cc:330] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: stopping tablet replica
I20260810 04:16:13.541684 18717 tablet_replica.cc:330] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: stopping tablet replica
I20260810 04:16:13.541738 18717 raft_consensus.cc:2229] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:13.541744 18716 raft_consensus.cc:2229] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:13.541792 18716 raft_consensus.cc:2258] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:13.541803 18717 raft_consensus.cc:2258] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:13.541775 18075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 04:16:13.541874 18715 tablet_replica.cc:330] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: stopping tablet replica
I20260810 04:16:13.541931 18715 raft_consensus.cc:2229] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 04:16:13.541934 18075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:13.541970 18715 raft_consensus.cc:2258] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:13.542305 18715 ts_tablet_manager.cc:1905] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:13.542464 18717 ts_tablet_manager.cc:1905] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:13.542455 18716 ts_tablet_manager.cc:1905] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:13.544014 18715 ts_tablet_manager.cc:1918] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:13.544013 18716 ts_tablet_manager.cc:1918] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:13.544078 18715 log.cc:1196] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/4ba7137d8e134ad7be04d3672296b3fe
I20260810 04:16:13.544078 18716 log.cc:1196] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/4ba7137d8e134ad7be04d3672296b3fe
I20260810 04:16:13.544312 18715 ts_tablet_manager.cc:1939] T 4ba7137d8e134ad7be04d3672296b3fe P 6a7d16f92878430ca0bd54598124613c: Deleting consensus metadata
I20260810 04:16:13.544312 18716 ts_tablet_manager.cc:1939] T 4ba7137d8e134ad7be04d3672296b3fe P 8fcbe54795fc48a68dd6a39b886d228e: Deleting consensus metadata
I20260810 04:16:13.544613 18058 catalog_manager.cc:4841] TS 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855): tablet 4ba7137d8e134ad7be04d3672296b3fe (table test-table [id=3faba631e7204e97a7f438f62fa311a3]) successfully deleted
I20260810 04:16:13.544771 18717 ts_tablet_manager.cc:1918] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:13.544791 18057 catalog_manager.cc:4841] TS 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125): tablet 4ba7137d8e134ad7be04d3672296b3fe (table test-table [id=3faba631e7204e97a7f438f62fa311a3]) successfully deleted
I20260810 04:16:13.544823 18717 log.cc:1196] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/4ba7137d8e134ad7be04d3672296b3fe
I20260810 04:16:13.545063 18717 ts_tablet_manager.cc:1939] T 4ba7137d8e134ad7be04d3672296b3fe P 97f5aa45ed7744f19f869ec1a1ad8092: Deleting consensus metadata
I20260810 04:16:13.545081 18527 tablet_service.cc:1467] Processing CreateTablet for tablet 300d675538034a6abc7ae6419a401bb6 (DEFAULT_TABLE table=test-table [id=6f09cb9cf0b34f2b8919531649082e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:13.545073 18379 tablet_service.cc:1467] Processing CreateTablet for tablet 300d675538034a6abc7ae6419a401bb6 (DEFAULT_TABLE table=test-table [id=6f09cb9cf0b34f2b8919531649082e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:13.545071 18232 tablet_service.cc:1467] Processing CreateTablet for tablet 300d675538034a6abc7ae6419a401bb6 (DEFAULT_TABLE table=test-table [id=6f09cb9cf0b34f2b8919531649082e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:13.545181 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300d675538034a6abc7ae6419a401bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:13.545189 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300d675538034a6abc7ae6419a401bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:13.545192 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300d675538034a6abc7ae6419a401bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:13.546224 18728 tablet_bootstrap.cc:492] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092: Bootstrap starting.
I20260810 04:16:13.546321 18059 catalog_manager.cc:4841] TS 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881): tablet 4ba7137d8e134ad7be04d3672296b3fe (table test-table [id=3faba631e7204e97a7f438f62fa311a3]) successfully deleted
I20260810 04:16:13.546446 18727 tablet_bootstrap.cc:492] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: Bootstrap starting.
I20260810 04:16:13.546446 18732 tablet_bootstrap.cc:492] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: Bootstrap starting.
I20260810 04:16:13.546551 18728 tablet_bootstrap.cc:654] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:13.546765 18727 tablet_bootstrap.cc:654] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:13.546765 18732 tablet_bootstrap.cc:654] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:13.547128 18728 tablet_bootstrap.cc:492] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092: No bootstrap required, opened a new log
I20260810 04:16:13.547183 18728 ts_tablet_manager.cc:1397] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:13.547292 18728 raft_consensus.cc:348] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.547340 18728 raft_consensus.cc:374] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:13.547353 18728 raft_consensus.cc:729] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f5aa45ed7744f19f869ec1a1ad8092, State: Initialized, Role: FOLLOWER
I20260810 04:16:13.547446 18727 tablet_bootstrap.cc:492] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: No bootstrap required, opened a new log
I20260810 04:16:13.547426 18728 consensus_queue.cc:260] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.547487 18727 ts_tablet_manager.cc:1397] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:13.547530 18728 ts_tablet_manager.cc:1428] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:13.547557 18732 tablet_bootstrap.cc:492] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: No bootstrap required, opened a new log
I20260810 04:16:13.547600 18732 ts_tablet_manager.cc:1397] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:13.547612 18727 raft_consensus.cc:348] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.547698 18727 raft_consensus.cc:374] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:13.547720 18727 raft_consensus.cc:729] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcbe54795fc48a68dd6a39b886d228e, State: Initialized, Role: FOLLOWER
I20260810 04:16:13.547720 18732 raft_consensus.cc:348] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.547789 18732 raft_consensus.cc:374] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:13.547778 18727 consensus_queue.cc:260] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.547821 18732 raft_consensus.cc:729] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7d16f92878430ca0bd54598124613c, State: Initialized, Role: FOLLOWER
I20260810 04:16:13.547879 18727 ts_tablet_manager.cc:1428] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:13.547874 18732 consensus_queue.cc:260] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.547974 18732 ts_tablet_manager.cc:1428] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:13.712563 18734 raft_consensus.cc:482] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:13.712682 18734 raft_consensus.cc:504] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.712918 18734 leader_election.cc:288] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:13.713209 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300d675538034a6abc7ae6419a401bb6" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e" is_pre_election: true
I20260810 04:16:13.713209 18253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300d675538034a6abc7ae6419a401bb6" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" is_pre_election: true
I20260810 04:16:13.713331 18550 raft_consensus.cc:2454] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 0.
I20260810 04:16:13.713331 18253 raft_consensus.cc:2454] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 0.
I20260810 04:16:13.713501 18331 leader_election.cc:302] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 97f5aa45ed7744f19f869ec1a1ad8092; no voters:
I20260810 04:16:13.713626 18734 raft_consensus.cc:2781] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:13.713686 18734 raft_consensus.cc:482] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:13.713719 18734 raft_consensus.cc:3037] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:13.714511 18734 raft_consensus.cc:504] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.714689 18734 leader_election.cc:288] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 election: Requested vote from peers 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881), 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855)
I20260810 04:16:13.714856 18550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300d675538034a6abc7ae6419a401bb6" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcbe54795fc48a68dd6a39b886d228e"
I20260810 04:16:13.714951 18550 raft_consensus.cc:3037] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:13.714905 18253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300d675538034a6abc7ae6419a401bb6" candidate_uuid: "6a7d16f92878430ca0bd54598124613c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5aa45ed7744f19f869ec1a1ad8092"
I20260810 04:16:13.715000 18253 raft_consensus.cc:3037] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:13.715617 18550 raft_consensus.cc:2454] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 1.
I20260810 04:16:13.715683 18253 raft_consensus.cc:2454] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a7d16f92878430ca0bd54598124613c in term 1.
I20260810 04:16:13.715835 18330 leader_election.cc:302] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a7d16f92878430ca0bd54598124613c, 8fcbe54795fc48a68dd6a39b886d228e; no voters:
I20260810 04:16:13.715962 18734 raft_consensus.cc:2781] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:13.716117 18734 raft_consensus.cc:686] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 1 LEADER]: Becoming Leader. State: Replica: 6a7d16f92878430ca0bd54598124613c, State: Running, Role: LEADER
I20260810 04:16:13.716190 18734 consensus_queue.cc:237] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } }
I20260810 04:16:13.716944 18076 catalog_manager.cc:5495] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7d16f92878430ca0bd54598124613c (127.15.216.66). New cstate: current_term: 1 leader_uuid: "6a7d16f92878430ca0bd54598124613c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a7d16f92878430ca0bd54598124613c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 } health_report { overall_health: UNKNOWN } } }
I20260810 04:16:13.737488 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:13.735174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.15.216.66:38125)
0810 04:16:13.735375 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:13.735377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:13.735384 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:13.735405 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:13.735405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:13.735416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:13.735475 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:13.735576 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:13.736065 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:13.736867 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:13.737070 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:13.737127 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:13.737232 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:13.737258 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:13.737293 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:13.737325 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 04:16:13.737345 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":124,"threads_started":1}
I20260810 04:16:13.740056 18253 raft_consensus.cc:1261] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092 [term 1 FOLLOWER]: Refusing update from remote peer 6a7d16f92878430ca0bd54598124613c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:13.740056 18550 raft_consensus.cc:1261] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Refusing update from remote peer 6a7d16f92878430ca0bd54598124613c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:13.740274 18745 consensus_queue.cc:1035] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f5aa45ed7744f19f869ec1a1ad8092" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:13.740396 18734 consensus_queue.cc:1035] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcbe54795fc48a68dd6a39b886d228e" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:16:13.745169 18696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 04:16:15.264788 18075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:15.264912 18075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:15.265817 18075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 998c7f08b19c4ddd81243cce7baef069: Sending DeleteTablet for 3 replicas of tablet 300d675538034a6abc7ae6419a401bb6
I20260810 04:16:15.266104 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet 300d675538034a6abc7ae6419a401bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:15 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:57592
I20260810 04:16:15.266172 18379 tablet_service.cc:1514] Processing DeleteTablet for tablet 300d675538034a6abc7ae6419a401bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:15 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:40756
I20260810 04:16:15.266170 18526 tablet_service.cc:1514] Processing DeleteTablet for tablet 300d675538034a6abc7ae6419a401bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:15 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:59894
I20260810 04:16:15.266294 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 18159
I20260810 04:16:15.266328 18775 tablet_replica.cc:330] T 300d675538034a6abc7ae6419a401bb6 P 97f5aa45ed7744f19f869ec1a1ad8092: stopping tablet replica
I20260810 04:16:15.273720 18776 tablet_replica.cc:330] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: stopping tablet replica
I20260810 04:16:15.273818 18776 raft_consensus.cc:2229] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:15.273864 18776 raft_consensus.cc:2258] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:15.273864 18777 tablet_replica.cc:330] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: stopping tablet replica
I20260810 04:16:15.273939 18777 raft_consensus.cc:2229] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:15.274015 18777 raft_consensus.cc:2258] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:15.274163 18776 ts_tablet_manager.cc:1905] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:15.274276 18777 ts_tablet_manager.cc:1905] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:15.275743 18776 ts_tablet_manager.cc:1918] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:16:15.275825 18776 log.cc:1196] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/300d675538034a6abc7ae6419a401bb6
I20260810 04:16:15.276091 18776 ts_tablet_manager.cc:1939] T 300d675538034a6abc7ae6419a401bb6 P 8fcbe54795fc48a68dd6a39b886d228e: Deleting consensus metadata
I20260810 04:16:15.276161 18777 ts_tablet_manager.cc:1918] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 04:16:15.276227 18777 log.cc:1196] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/300d675538034a6abc7ae6419a401bb6
I20260810 04:16:15.276508 18777 ts_tablet_manager.cc:1939] T 300d675538034a6abc7ae6419a401bb6 P 6a7d16f92878430ca0bd54598124613c: Deleting consensus metadata
I20260810 04:16:15.276575 18058 catalog_manager.cc:4841] TS 8fcbe54795fc48a68dd6a39b886d228e (127.15.216.67:34855): tablet 300d675538034a6abc7ae6419a401bb6 (table test-table [id=6f09cb9cf0b34f2b8919531649082e57]) successfully deleted
I20260810 04:16:15.276835 18057 catalog_manager.cc:4841] TS 6a7d16f92878430ca0bd54598124613c (127.15.216.66:38125): tablet 300d675538034a6abc7ae6419a401bb6 (table test-table [id=6f09cb9cf0b34f2b8919531649082e57]) successfully deleted
W20260810 04:16:15.280894 18059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:33881 (error 108)
I20260810 04:16:15.281152 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 18312
I20260810 04:16:15.283504 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:15.281316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33881 (local address 127.0.0.1:57612)
0810 04:16:15.283420 (+ 2104us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:15.283468 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2079,"thread_start_us":60,"threads_started":1}
W20260810 04:16:15.283628 18059 catalog_manager.cc:4584] TS 97f5aa45ed7744f19f869ec1a1ad8092 (127.15.216.65:33881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 300d675538034a6abc7ae6419a401bb6: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33881: connect: Connection refused (error 111)
I20260810 04:16:15.292917 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 18457
I20260810 04:16:15.302564 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 18038
2026-08-10T04:16:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6194 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:15 dist-test-slave-g350 krb5kdc[18782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:15 dist-test-slave-g350 krb5kdc[18782](info): set up 2 sockets
Aug 10 04:16:15 dist-test-slave-g350 krb5kdc[18782](info): commencing operation
krb5kdc: starting...
W20260810 04:16:16.386409 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:16 dist-test-slave-g350 krb5kdc[18782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:16.516894 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:33107
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:33107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:16.599841 18795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:16.600047 18795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:16.600069 18795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:16.601117 18795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:16.601156 18795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:16.601171 18795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:16.601183 18795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:16.601195 18795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:16.602635 18795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:33107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:33107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:16.602830 18795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:16.602993 18795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:16.607831 18803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:16.607882 18805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:16.607848 18802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:16.608099 18795 server_base.cc:1034] running on GCE node
I20260810 04:16:16.608325 18795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:16.608594 18795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:16.609732 18795 hybrid_clock.cc:648] HybridClock initialized: now 1786335376609726 us; error 37 us; skew 500 ppm
Aug 10 04:16:16 dist-test-slave-g350 krb5kdc[18782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:16.617280 18795 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:16.617805 18795 webserver.cc:460] Webserver started at http://127.15.216.126:36151/ using document root <none> and password file <none>
I20260810 04:16:16.618007 18795 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:16.618058 18795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:16.618148 18795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:16.619167 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "88695b8bcf04474eb06479010d41fe17"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "3f0c9db32239926848dbe3a3492362b3"
server_key_iv: "e0f2dc49bfd85e44a2a8a53ddc0579d4"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.619479 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "88695b8bcf04474eb06479010d41fe17"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "3f0c9db32239926848dbe3a3492362b3"
server_key_iv: "e0f2dc49bfd85e44a2a8a53ddc0579d4"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.620759 18795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:16.621538 18813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:16.621803 18795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:16.621922 18795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "88695b8bcf04474eb06479010d41fe17"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "3f0c9db32239926848dbe3a3492362b3"
server_key_iv: "e0f2dc49bfd85e44a2a8a53ddc0579d4"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.622010 18795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:16.635082 18795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:16.637030 18795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:16.637148 18795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:16.641180 18795 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:33107
I20260810 04:16:16.641237 18872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:33107 every 8 connection(s)
I20260810 04:16:16.641667 18795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:16.642362 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 18795
I20260810 04:16:16.642438 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:16.642649 16225 external_mini_cluster.cc:1428] Setting key 1526b7990813b84262f1c98963094899
I20260810 04:16:16.642656 18873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:16.646345 18873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17: Bootstrap starting.
I20260810 04:16:16.647094 18873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:16.647428 18873 log.cc:824] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:16.648557 18873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17: No bootstrap required, opened a new log
I20260810 04:16:16.650933 18873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88695b8bcf04474eb06479010d41fe17" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33107 } }
I20260810 04:16:16.651139 18873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:16.651199 18873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88695b8bcf04474eb06479010d41fe17, State: Initialized, Role: FOLLOWER
I20260810 04:16:16.651331 18873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88695b8bcf04474eb06479010d41fe17" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33107 } }
I20260810 04:16:16.651432 18873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:16.651477 18873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:16.651526 18873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 04:16:16 dist-test-slave-g350 krb5kdc[18782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:16.652256 18873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88695b8bcf04474eb06479010d41fe17" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33107 } }
I20260810 04:16:16.652380 18873 leader_election.cc:302] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88695b8bcf04474eb06479010d41fe17; no voters:
I20260810 04:16:16.652557 18873 leader_election.cc:288] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:16.652738 18880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:16.652840 18873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:16.652916 18880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [term 1 LEADER]: Becoming Leader. State: Replica: 88695b8bcf04474eb06479010d41fe17, State: Running, Role: LEADER
I20260810 04:16:16.653090 18880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88695b8bcf04474eb06479010d41fe17" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33107 } }
I20260810 04:16:16.653403 18880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88695b8bcf04474eb06479010d41fe17. Latest consensus state: current_term: 1 leader_uuid: "88695b8bcf04474eb06479010d41fe17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88695b8bcf04474eb06479010d41fe17" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33107 } } }
I20260810 04:16:16.653523 18880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:16.653787 18888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:16.653894 18879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88695b8bcf04474eb06479010d41fe17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88695b8bcf04474eb06479010d41fe17" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33107 } } }
I20260810 04:16:16.653990 18879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:16.654510 18888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:16.656167 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:16.643186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.15.216.126:33107)
0810 04:16:16.643402 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:16.643412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:16.643446 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:16:16.644041 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:16.644050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:16.644112 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:16.644263 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:16.644918 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:16.646292 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:16.647230 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:16.647763 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:16.647894 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:16.652129 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:16.652132 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:16.652133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:16.652144 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:16.654431 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:16.654878 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:16.654882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:16.654882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:16.654989 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:16.655245 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:16.655246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:16.655247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:16.655527 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:16.655680 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:16.655798 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 04:16:16.655952 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260810 04:16:16.656494 18888 catalog_manager.cc:1347] Generated new cluster ID: df1487e72e8542d5b9f1041ba914961f
I20260810 04:16:16.656565 18888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:16.670186 18888 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:16.670747 18888 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:16.675626 18888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88695b8bcf04474eb06479010d41fe17: Generated new TSK 0
I20260810 04:16:16.675832 18888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:16.705658 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33107
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:16.793557 18900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:16.793812 18900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:16.793839 18900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:16.795231 18900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:16.795275 18900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:16.795305 18900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:16.796949 18900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:16.797168 18900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:16.797375 18900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:16.802524 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:16.802539 18909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:16.802543 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:16.802968 18900 server_base.cc:1034] running on GCE node
I20260810 04:16:16.803203 18900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:16.803465 18900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:16.804641 18900 hybrid_clock.cc:648] HybridClock initialized: now 1786335376804632 us; error 38 us; skew 500 ppm
Aug 10 04:16:16 dist-test-slave-g350 krb5kdc[18782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:16.807591 18900 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:16.808092 18900 webserver.cc:460] Webserver started at http://127.15.216.65:37233/ using document root <none> and password file <none>
I20260810 04:16:16.808377 18900 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:16.808449 18900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:16.808607 18900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:16.809772 18900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "3c141bffd8894d1aab6f425f15ae3a6b"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "d6158e8a500a22be3fc04e296e447480"
server_key_iv: "cdeb8742775b317f376bb9009930a540"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.810135 18900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "3c141bffd8894d1aab6f425f15ae3a6b"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "d6158e8a500a22be3fc04e296e447480"
server_key_iv: "cdeb8742775b317f376bb9009930a540"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.811759 18900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:16.812669 18917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:16.812945 18900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:16.813083 18900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "3c141bffd8894d1aab6f425f15ae3a6b"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "d6158e8a500a22be3fc04e296e447480"
server_key_iv: "cdeb8742775b317f376bb9009930a540"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.813174 18900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:16.837611 18900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:16.840149 18900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:16.840346 18900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:16.840682 18900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:16.841068 18900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:16.841137 18900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:16.841209 18900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:16.841243 18900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:16.848487 18900 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:37127
I20260810 04:16:16.848536 19041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:37127 every 8 connection(s)
I20260810 04:16:16.848944 18900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:16.851863 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 18900
I20260810 04:16:16.851974 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:16.852181 16225 external_mini_cluster.cc:1428] Setting key fc3fa4a07a20089415ea6403446e5eaa
Aug 10 04:16:16 dist-test-slave-g350 krb5kdc[18782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:16.860881 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:16.849758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:37769 (local address 127.15.216.126:33107)
0810 04:16:16.849833 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:16.849835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:16.850103 (+ 268us) server_negotiation.cc:407] Connection header received
0810 04:16:16.850700 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:16.850702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:16.850717 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:16.850774 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:16.851951 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:16.852690 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:16.854424 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:16.854638 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:16.854855 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:16.857344 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:16.857345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:16.857346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:16.857354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:16.859198 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:16.859649 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:16.859650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:16.859650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:16.859809 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:16.860113 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:16.860114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:16.860115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:16.860277 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:16.860357 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:16.860646 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 04:16:16.860746 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:16:16.860965 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:16.849741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:33107 (local address 127.15.216.65:37769)
0810 04:16:16.849968 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:16.850008 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:16.850573 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:16.850857 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:16.850861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:16.851499 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:16.851801 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:16.851810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:16.852792 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:16.852793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:16.854312 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:16.854316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:16.854929 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:16.854930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:16.855041 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:16.855640 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:16.855649 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:16.857239 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:16.859295 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:16.859303 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:16.859305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:16.859562 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:16.859915 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:16.859916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:16.859916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:16.860035 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:16.860353 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:16.860361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:16.860501 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 04:16:16.860594 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 04:16:16.860728 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":135,"threads_started":1}
I20260810 04:16:16.861631 19042 heartbeater.cc:347] Connected to a master server at 127.15.216.126:33107
I20260810 04:16:16.861768 19042 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:16.861984 19042 heartbeater.cc:511] Master 127.15.216.126:33107 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
I20260810 04:16:16.862625 18833 ts_manager.cc:194] Registered new tserver with Master: 3c141bffd8894d1aab6f425f15ae3a6b (127.15.216.65:37127)
I20260810 04:16:16.863322 18833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:37769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:16.884475 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33107
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:16.972177 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:16.972404 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:16.972427 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:16.973652 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:16.973696 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:16.973723 19047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:16.975183 19047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:16.975440 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:16.975672 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:16.980980 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:16.981005 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:16.981072 19047 server_base.cc:1034] running on GCE node
W20260810 04:16:16.981027 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:16.981470 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:16.981747 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:16.982923 19047 hybrid_clock.cc:648] HybridClock initialized: now 1786335376982907 us; error 30 us; skew 500 ppm
Aug 10 04:16:16 dist-test-slave-g350 krb5kdc[18782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:16.985482 19047 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:16.986025 19047 webserver.cc:460] Webserver started at http://127.15.216.66:36619/ using document root <none> and password file <none>
I20260810 04:16:16.986263 19047 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:16.986341 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:16.986495 19047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:16.987577 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "23a0a537a4494445b02ea9a7571aa306"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "95f0bc6a672093d7496b4c2555c29973"
server_key_iv: "fa2a7870a56cb052450a8401242a2828"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.987996 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "23a0a537a4494445b02ea9a7571aa306"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "95f0bc6a672093d7496b4c2555c29973"
server_key_iv: "fa2a7870a56cb052450a8401242a2828"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.989434 19047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:16.990374 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:16.990583 19047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:16.990705 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "23a0a537a4494445b02ea9a7571aa306"
format_stamp: "Formatted at 2026-08-10 04:16:16 on dist-test-slave-g350"
server_key: "95f0bc6a672093d7496b4c2555c29973"
server_key_iv: "fa2a7870a56cb052450a8401242a2828"
server_key_version: "encryptionkey@0"
I20260810 04:16:16.990783 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:17.025846 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:17.028333 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:17.028488 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:17.028769 19047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:17.029099 19047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:17.029134 19047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:17.029157 19047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:17.029203 19047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:17.035441 19047 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:42019
I20260810 04:16:17.035529 19185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:42019 every 8 connection(s)
I20260810 04:16:17.035905 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:17.040907 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 19047
I20260810 04:16:17.040989 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:17.041174 16225 external_mini_cluster.cc:1428] Setting key bfda96404d0ab9fd6341660f7fe8b359
Aug 10 04:16:17 dist-test-slave-g350 krb5kdc[18782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335376, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:17.045926 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:17.036690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:39083 (local address 127.15.216.126:33107)
0810 04:16:17.036746 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:17.036748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:17.036926 (+ 178us) server_negotiation.cc:407] Connection header received
0810 04:16:17.037420 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:17.037421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:17.037432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:17.037493 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:17.038407 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.038896 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.039934 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.040127 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.040184 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:17.042726 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:17.042727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:17.042728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:17.042742 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:17.044463 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:17.044875 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:17.044877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:17.044877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:17.044944 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:17.045219 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:17.045220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:17.045220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:17.045365 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:17.045422 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:17.045768 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 04:16:17.045863 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:16:17.046315 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:17.036628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:33107 (local address 127.15.216.66:39083)
0810 04:16:17.036789 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:17.036831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:17.037334 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:17.037543 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:17.037546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:17.038123 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:17.038326 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:17.038331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.038989 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.038990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:17.039846 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:17.039848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.040208 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.040209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:17.040350 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:17.040915 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:17.040931 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:17.042616 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:17.044554 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:17.044561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:17.044563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:17.044797 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:17.045009 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:17.045010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:17.045010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:17.045152 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:17.045436 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:17.045443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:17.045629 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 04:16:17.045769 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 04:16:17.045954 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260810 04:16:17.046849 19186 heartbeater.cc:347] Connected to a master server at 127.15.216.126:33107
I20260810 04:16:17.046960 19186 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:17.047169 19186 heartbeater.cc:511] Master 127.15.216.126:33107 requested a full tablet report, sending...
I20260810 04:16:17.047544 18833 ts_manager.cc:194] Registered new tserver with Master: 23a0a537a4494445b02ea9a7571aa306 (127.15.216.66:42019)
I20260810 04:16:17.047923 18833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:39083
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:17.072597 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33107
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:17.156992 19191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:17.157212 19191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:17.157238 19191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:17.158509 19191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:17.158553 19191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:17.158581 19191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:17.160032 19191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:17.160282 19191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:17.160526 19191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:17.165906 19197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:17.165915 19199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:17.165915 19196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:17.166396 19191 server_base.cc:1034] running on GCE node
I20260810 04:16:17.166652 19191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:17.166926 19191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:17.168099 19191 hybrid_clock.cc:648] HybridClock initialized: now 1786335377168095 us; error 38 us; skew 500 ppm
Aug 10 04:16:17 dist-test-slave-g350 krb5kdc[18782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335377, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:17.170790 19191 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:17.171203 19191 webserver.cc:460] Webserver started at http://127.15.216.67:35479/ using document root <none> and password file <none>
I20260810 04:16:17.171411 19191 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:17.171451 19191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:17.171540 19191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:17.172544 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "c1edb0c57f1342618ec60156c2344303"
format_stamp: "Formatted at 2026-08-10 04:16:17 on dist-test-slave-g350"
server_key: "d940692128a792ef7c4d36cf9599a8a7"
server_key_iv: "3a484122b056b3a7a430a3b949387156"
server_key_version: "encryptionkey@0"
I20260810 04:16:17.172839 19191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "c1edb0c57f1342618ec60156c2344303"
format_stamp: "Formatted at 2026-08-10 04:16:17 on dist-test-slave-g350"
server_key: "d940692128a792ef7c4d36cf9599a8a7"
server_key_iv: "3a484122b056b3a7a430a3b949387156"
server_key_version: "encryptionkey@0"
I20260810 04:16:17.174268 19191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:17.175104 19206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:17.175407 19191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:17.175544 19191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "c1edb0c57f1342618ec60156c2344303"
format_stamp: "Formatted at 2026-08-10 04:16:17 on dist-test-slave-g350"
server_key: "d940692128a792ef7c4d36cf9599a8a7"
server_key_iv: "3a484122b056b3a7a430a3b949387156"
server_key_version: "encryptionkey@0"
I20260810 04:16:17.175633 19191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:17.211146 19191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:17.213596 19191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:17.213730 19191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:17.213937 19191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:17.214282 19191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:17.214318 19191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:17.214372 19191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:17.214406 19191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:17.220884 19191 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:39047
I20260810 04:16:17.220938 19331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:39047 every 8 connection(s)
I20260810 04:16:17.221277 19191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:16:17 dist-test-slave-g350 krb5kdc[18782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335377, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:17.228813 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 19191
I20260810 04:16:17.228926 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:17.229148 16225 external_mini_cluster.cc:1428] Setting key f36a430b028db8c556671ce5bfb3828d
I20260810 04:16:17.232025 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:17.222089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:51415 (local address 127.15.216.126:33107)
0810 04:16:17.222163 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:17.222165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:17.222384 (+ 219us) server_negotiation.cc:407] Connection header received
0810 04:16:17.222907 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:17.222908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:17.222919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:17.222970 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:17.224070 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.224652 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.225889 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.226087 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.226146 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:17.228584 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:17.228586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:17.228586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:17.228594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:17.230572 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:17.231014 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:17.231016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:17.231016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:17.231111 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:17.231402 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:17.231403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:17.231403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:17.231536 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:17.231610 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:17.231878 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 04:16:17.231955 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:16:17.232175 19334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:17.222060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:33107 (local address 127.15.216.67:51415)
0810 04:16:17.222249 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:17.222295 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:17.222815 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:17.223041 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:17.223044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:17.223750 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:17.223961 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:17.223967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.224768 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.224769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:17.225806 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:17.225809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:17.226165 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:17.226166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:17.226310 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:17.226800 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:17.226809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:17.228475 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:17.230658 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:17.230665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:17.230667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:17.230922 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:17.231186 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:17.231187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:17.231187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:17.231313 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:17.231607 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:17.231612 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:17.231728 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 04:16:17.231830 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 04:16:17.231957 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260810 04:16:17.232714 19332 heartbeater.cc:347] Connected to a master server at 127.15.216.126:33107
I20260810 04:16:17.232802 19332 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:17.232999 19332 heartbeater.cc:511] Master 127.15.216.126:33107 requested a full tablet report, sending...
I20260810 04:16:17.233477 18833 ts_manager.cc:194] Registered new tserver with Master: c1edb0c57f1342618ec60156c2344303 (127.15.216.67:39047)
I20260810 04:16:17.233883 18833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:51415
I20260810 04:16:17.241336 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:16:17.255656 18877 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:17.254617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.15.216.126:33107)
0810 04:16:17.254684 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:17.254685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:17.254770 (+ 85us) server_negotiation.cc:407] Connection header received
0810 04:16:17.254818 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:17.254818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:17.254829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:17.254893 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:17.255574 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42656: BlockingRecv error: recv got EOF from 127.0.0.1:42656 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:16:17.256685 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 18900
I20260810 04:16:17.265069 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 19047
I20260810 04:16:17.271186 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 19191
I20260810 04:16:17.278637 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 18795
2026-08-10T04:16:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1976 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:17 dist-test-slave-g350 krb5kdc[19344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:17 dist-test-slave-g350 krb5kdc[19344](info): set up 2 sockets
Aug 10 04:16:17 dist-test-slave-g350 krb5kdc[19344](info): commencing operation
krb5kdc: starting...
W20260810 04:16:18.375283 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:18 dist-test-slave-g350 krb5kdc[19344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:18.507104 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41677
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:18.594368 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:18.594586 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:18.594610 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:18.595803 19357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:18.595842 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:18.595858 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:18.595872 19357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:18.595886 19357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:18.597569 19357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:18.597808 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:18.597991 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:18.603600 19364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:18.603679 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:18.603612 19366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:18.603821 19357 server_base.cc:1034] running on GCE node
I20260810 04:16:18.604163 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:18.604491 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:18.605702 19357 hybrid_clock.cc:648] HybridClock initialized: now 1786335378605666 us; error 81 us; skew 500 ppm
Aug 10 04:16:18 dist-test-slave-g350 krb5kdc[19344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335378, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:18.608584 19357 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:18.609053 19357 webserver.cc:460] Webserver started at http://127.15.216.126:32987/ using document root <none> and password file <none>
I20260810 04:16:18.609349 19357 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:18.609427 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:18.609624 19357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:18.610821 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "ec0a3e3cf94f4f2886181725b5b3b18b"
format_stamp: "Formatted at 2026-08-10 04:16:18 on dist-test-slave-g350"
server_key: "e8c13cf36759dc05b4e2a9e0b2027401"
server_key_iv: "cf68d30e5f4ff764cac1ae6eb1179868"
server_key_version: "encryptionkey@0"
I20260810 04:16:18.611232 19357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "ec0a3e3cf94f4f2886181725b5b3b18b"
format_stamp: "Formatted at 2026-08-10 04:16:18 on dist-test-slave-g350"
server_key: "e8c13cf36759dc05b4e2a9e0b2027401"
server_key_iv: "cf68d30e5f4ff764cac1ae6eb1179868"
server_key_version: "encryptionkey@0"
I20260810 04:16:18.612744 19357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:18.613651 19373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:18.613869 19357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:18.613988 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "ec0a3e3cf94f4f2886181725b5b3b18b"
format_stamp: "Formatted at 2026-08-10 04:16:18 on dist-test-slave-g350"
server_key: "e8c13cf36759dc05b4e2a9e0b2027401"
server_key_iv: "cf68d30e5f4ff764cac1ae6eb1179868"
server_key_version: "encryptionkey@0"
I20260810 04:16:18.614082 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:18.629293 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:18.631521 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:18.631699 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:18.636530 19357 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41677
I20260810 04:16:18.636565 19427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41677 every 8 connection(s)
I20260810 04:16:18.637166 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:18.637948 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:18.640305 19428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b: Bootstrap starting.
I20260810 04:16:18.640878 19428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:18.641124 19428 log.cc:824] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:18.641855 19428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b: No bootstrap required, opened a new log
I20260810 04:16:18.643226 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 19357
I20260810 04:16:18.643349 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:18.643544 16225 external_mini_cluster.cc:1428] Setting key c2eb16d94d73f62f9ec883ca98285e2b
I20260810 04:16:18.643512 19428 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41677 } }
I20260810 04:16:18.643684 19428 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:18.643766 19428 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0a3e3cf94f4f2886181725b5b3b18b, State: Initialized, Role: FOLLOWER
I20260810 04:16:18.643918 19428 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41677 } }
I20260810 04:16:18.644001 19428 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:18.644035 19428 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:18.644081 19428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:18.644855 19428 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41677 } }
I20260810 04:16:18.644950 19428 leader_election.cc:302] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0a3e3cf94f4f2886181725b5b3b18b; no voters:
I20260810 04:16:18.645176 19428 leader_election.cc:288] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:18.645208 19433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:18.645327 19433 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [term 1 LEADER]: Becoming Leader. State: Replica: ec0a3e3cf94f4f2886181725b5b3b18b, State: Running, Role: LEADER
I20260810 04:16:18.645401 19433 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41677 } }
I20260810 04:16:18.645440 19428 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:18.645771 19435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41677 } } }
I20260810 04:16:18.645864 19435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:18.646459 19440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:18.647068 19440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:18.646065 19434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0a3e3cf94f4f2886181725b5b3b18b. Latest consensus state: current_term: 1 leader_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0a3e3cf94f4f2886181725b5b3b18b" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41677 } } }
I20260810 04:16:18.647950 19434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:18.649134 19440 catalog_manager.cc:1347] Generated new cluster ID: b5e503d32f7648019da974abcffe62dd
I20260810 04:16:18.649188 19440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:16:18 dist-test-slave-g350 krb5kdc[19344](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:18.656363 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:18.644062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.15.216.126:41677)
0810 04:16:18.644750 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:18.644759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:18.644788 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:16:18.644886 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:18.644894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:18.644995 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:18.645187 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:18.646147 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:18.648072 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:18.649036 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:18.649646 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:18.649753 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:18.652509 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:18.652514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:18.652514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:18.652525 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:18.654521 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:18.654904 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:18.654907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:18.654907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:18.655002 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:18.655172 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:18.655173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:18.655173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:18.655444 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:18.655590 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:18.656004 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 04:16:18.656139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":563,"threads_started":1}
I20260810 04:16:18.657343 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:16:18.664381 19440 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:18.665138 19440 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:18.671249 19440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0a3e3cf94f4f2886181725b5b3b18b: Generated new TSK 0
I20260810 04:16:18.671442 19440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:16:18.676693 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:18.672475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.15.216.126:41677)
0810 04:16:18.672541 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:18.672543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:18.672697 (+ 154us) server_negotiation.cc:407] Connection header received
0810 04:16:18.672792 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:18.672793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:18.672804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:18.672853 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:18.673305 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:18.673870 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:18.675287 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:18.675481 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:18.675540 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:18.676263 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:18.676264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:18.676265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:18.676459 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:18.676475 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:18.676481 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:18.676529 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:18.676619 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 04:16:18.676641 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:16:18.679047 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 19357
2026-08-10T04:16:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1399 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:18 dist-test-slave-g350 krb5kdc[19467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:18 dist-test-slave-g350 krb5kdc[19467](info): set up 2 sockets
Aug 10 04:16:18 dist-test-slave-g350 krb5kdc[19467](info): commencing operation
krb5kdc: starting...
W20260810 04:16:19.766350 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:19 dist-test-slave-g350 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:19.904379 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:38281
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:19.992007 19482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:19.992210 19482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:19.992233 19482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:19.993319 19482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:19.993355 19482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:19.993369 19482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:19.993382 19482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:19.993400 19482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:19.994874 19482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:38281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:19.995076 19482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:19.995244 19482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:20.000494 19491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.000520 19487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.000494 19488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:20.000934 19482 server_base.cc:1034] running on GCE node
I20260810 04:16:20.001210 19482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:20.001541 19482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:20.002748 19482 hybrid_clock.cc:648] HybridClock initialized: now 1786335380002730 us; error 32 us; skew 500 ppm
Aug 10 04:16:19 dist-test-slave-g350 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335379, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:20.005743 19482 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:20.006363 19482 webserver.cc:460] Webserver started at http://127.15.216.126:38883/ using document root <none> and password file <none>
I20260810 04:16:20.006629 19482 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:20.006702 19482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:20.006871 19482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:20.008013 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "a18c17298ca8cc6c97c1b5b24df5a452"
server_key_iv: "95f2b08d8f7b000fa36930ee928b544e"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.008395 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "a18c17298ca8cc6c97c1b5b24df5a452"
server_key_iv: "95f2b08d8f7b000fa36930ee928b544e"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.010056 19482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:20.011173 19498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.011438 19482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:20.011560 19482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "a18c17298ca8cc6c97c1b5b24df5a452"
server_key_iv: "95f2b08d8f7b000fa36930ee928b544e"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.011656 19482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:20.025220 19482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:20.027540 19482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:20.027720 19482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:20.032128 19482 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:38281
I20260810 04:16:20.032169 19556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:38281 every 8 connection(s)
I20260810 04:16:20.032550 19482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:20.033720 19557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:20.036131 19557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49: Bootstrap starting.
I20260810 04:16:20.036712 19557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:20.037019 19557 log.cc:824] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:20.037834 19557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49: No bootstrap required, opened a new log
I20260810 04:16:20.039507 19557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38281 } }
I20260810 04:16:20.039652 19557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:20.039703 19557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d35c668fc8e47bdb75b44bc4a5a7a49, State: Initialized, Role: FOLLOWER
I20260810 04:16:20.039865 19557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38281 } }
I20260810 04:16:20.039928 19557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:20.039950 19557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:20.040002 19557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:20.040259 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 19482
I20260810 04:16:20.040343 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:20.040529 16225 external_mini_cluster.cc:1428] Setting key 8ba63d03a682e646bdeb9f9867df8e78
I20260810 04:16:20.041136 19557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38281 } }
I20260810 04:16:20.041317 19557 leader_election.cc:302] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d35c668fc8e47bdb75b44bc4a5a7a49; no voters:
I20260810 04:16:20.041500 19557 leader_election.cc:288] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:20.041553 19562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:20.041778 19557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:20.041790 19562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [term 1 LEADER]: Becoming Leader. State: Replica: 2d35c668fc8e47bdb75b44bc4a5a7a49, State: Running, Role: LEADER
I20260810 04:16:20.041990 19562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38281 } }
I20260810 04:16:20.042408 19563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38281 } } }
I20260810 04:16:20.042510 19563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:20.042848 19564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d35c668fc8e47bdb75b44bc4a5a7a49. Latest consensus state: current_term: 1 leader_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d35c668fc8e47bdb75b44bc4a5a7a49" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38281 } } }
I20260810 04:16:20.042948 19564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:20.043367 19568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:20.043875 19568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:20.045432 19568 catalog_manager.cc:1347] Generated new cluster ID: da070382bf814c43b51f1965bd9a6309
I20260810 04:16:20.045485 19568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:20.055174 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.041070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.15.216.126:38281)
0810 04:16:20.041300 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:20.041310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:20.041347 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:16:20.041468 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:20.041476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:20.041535 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:20.042472 (+ 937us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:20.042573 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.044418 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.045521 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.046141 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.046270 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.049190 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:20.049195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:20.049195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:20.049209 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:20.053139 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.053634 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.053638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.053638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.053780 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.054006 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.054007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.054007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.054350 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:20.054506 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:20.054623 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 04:16:20.054914 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":102,"threads_started":1}
I20260810 04:16:20.096767 19568 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:20.097528 19568 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:20.103652 19568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d35c668fc8e47bdb75b44bc4a5a7a49: Generated new TSK 0
I20260810 04:16:20.103868 19568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:20.138788 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38281
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:20.224503 19588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:20.224720 19588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:20.224746 19588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:20.225950 19588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:20.225994 19588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:20.226022 19588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:20.227519 19588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:20.227773 19588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:20.228044 19588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:20.233675 19593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.233640 19594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.233645 19596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:20.233670 19588 server_base.cc:1034] running on GCE node
I20260810 04:16:20.234170 19588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:20.234445 19588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:20.235618 19588 hybrid_clock.cc:648] HybridClock initialized: now 1786335380235605 us; error 49 us; skew 500 ppm
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:20.238420 19588 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:20.238933 19588 webserver.cc:460] Webserver started at http://127.15.216.65:41343/ using document root <none> and password file <none>
I20260810 04:16:20.239164 19588 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:20.239233 19588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:20.239399 19588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:20.240474 19588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "e601db31b834460986eaa8328b758650"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "c43b5d6ac43f59bd7ef81255b337f76d"
server_key_iv: "f5a0f7bd7e6054335dca265fc928daa9"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.240852 19588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "e601db31b834460986eaa8328b758650"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "c43b5d6ac43f59bd7ef81255b337f76d"
server_key_iv: "f5a0f7bd7e6054335dca265fc928daa9"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.242411 19588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:20.243208 19603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.243378 19588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:20.243489 19588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "e601db31b834460986eaa8328b758650"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "c43b5d6ac43f59bd7ef81255b337f76d"
server_key_iv: "f5a0f7bd7e6054335dca265fc928daa9"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.243564 19588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:20.260255 19588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:20.262835 19588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:20.263011 19588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:20.263336 19588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:20.263700 19588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:20.263758 19588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.263794 19588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:20.263844 19588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.270526 19588 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:37587
I20260810 04:16:20.270589 19724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:37587 every 8 connection(s)
I20260810 04:16:20.270949 19588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:20.275205 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 19588
I20260810 04:16:20.275348 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:20.275563 16225 external_mini_cluster.cc:1428] Setting key ee117740ee15739754d2387f991ddd47
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:20.281859 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.271873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:35173 (local address 127.15.216.126:38281)
0810 04:16:20.271974 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:20.271977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:20.272165 (+ 188us) server_negotiation.cc:407] Connection header received
0810 04:16:20.272762 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:20.272764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:20.272782 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:20.272820 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:20.273852 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.274378 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.275426 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.275766 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.275865 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.278398 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:20.278399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:20.278400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:20.278408 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:20.280155 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.280686 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.280688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.280688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.280759 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.281073 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.281074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.281074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.281238 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:20.281317 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:20.281642 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 04:16:20.281741 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:16:20.281909 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.271791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38281 (local address 127.15.216.65:35173)
0810 04:16:20.272024 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:20.272071 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:20.272646 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:20.272920 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:20.272924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:20.273513 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:20.273735 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:20.273740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.274484 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.274485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:20.275304 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:20.275307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.275881 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.275882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:20.276035 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.276648 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:20.276656 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:20.278282 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:20.280248 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:20.280256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:20.280258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:20.280594 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:20.280839 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:20.280840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:20.280840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:20.280996 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:20.281342 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:20.281347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:20.281469 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 04:16:20.281555 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:16:20.281742 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260810 04:16:20.282495 19725 heartbeater.cc:347] Connected to a master server at 127.15.216.126:38281
I20260810 04:16:20.282613 19725 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:20.282795 19725 heartbeater.cc:511] Master 127.15.216.126:38281 requested a full tablet report, sending...
I20260810 04:16:20.283327 19517 ts_manager.cc:194] Registered new tserver with Master: e601db31b834460986eaa8328b758650 (127.15.216.65:37587)
I20260810 04:16:20.283938 19517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:35173
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:20.306419 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38281
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:20.395499 19731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:20.395714 19731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:20.395744 19731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:20.396850 19731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:20.396888 19731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:20.396914 19731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:20.398460 19731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:20.398730 19731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:20.398975 19731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:20.404186 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.404335 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.404284 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:20.404387 19731 server_base.cc:1034] running on GCE node
I20260810 04:16:20.404661 19731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:20.404857 19731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:20.405997 19731 hybrid_clock.cc:648] HybridClock initialized: now 1786335380405983 us; error 30 us; skew 500 ppm
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:20.408480 19731 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:20.408972 19731 webserver.cc:460] Webserver started at http://127.15.216.66:40733/ using document root <none> and password file <none>
I20260810 04:16:20.409228 19731 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:20.409302 19731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:20.409474 19731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:20.410578 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "5058dae69ca646558fee8cc0e2c2bd4d"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "69389e1cace8af8cd2a234140c1d8d6f"
server_key_iv: "93b9a301761665fcf2a23cd3d3400967"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.410962 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "5058dae69ca646558fee8cc0e2c2bd4d"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "69389e1cace8af8cd2a234140c1d8d6f"
server_key_iv: "93b9a301761665fcf2a23cd3d3400967"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.412304 19731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:16:20.413102 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.413415 19731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:20.413514 19731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "5058dae69ca646558fee8cc0e2c2bd4d"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "69389e1cace8af8cd2a234140c1d8d6f"
server_key_iv: "93b9a301761665fcf2a23cd3d3400967"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.413626 19731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:20.437577 19731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:20.439827 19731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:20.439981 19731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:20.440246 19731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:20.440608 19731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:20.440672 19731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.440716 19731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:20.440771 19731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.447620 19731 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:46221
I20260810 04:16:20.447768 19872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:46221 every 8 connection(s)
I20260810 04:16:20.448118 19731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:20.453756 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 19731
I20260810 04:16:20.453903 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:20.454116 16225 external_mini_cluster.cc:1428] Setting key 4312b43686c285a6f8881e3e2637a745
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:20.457993 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.448951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:42275 (local address 127.15.216.126:38281)
0810 04:16:20.449000 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:20.449003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:20.449154 (+ 151us) server_negotiation.cc:407] Connection header received
0810 04:16:20.449695 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:20.449697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:20.449709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:20.449773 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:20.450699 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.451203 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.452218 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.452417 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.452482 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.455085 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:20.455086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:20.455086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:20.455093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:20.456392 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.456900 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.456901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.456902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.456974 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.457216 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.457217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.457217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.457374 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:20.457450 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:20.457809 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 04:16:20.457910 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:16:20.458202 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.448854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38281 (local address 127.15.216.66:42275)
0810 04:16:20.449025 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:20.449069 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:20.449564 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:20.449851 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:20.449854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:20.450421 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:20.450614 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:20.450619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.451284 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.451285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:20.452102 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:20.452105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.452502 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.452502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:20.452607 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.453164 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:20.453173 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:20.454984 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:20.456471 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:20.456479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:20.456481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:20.456814 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:20.457045 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:20.457046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:20.457046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:20.457155 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:20.457467 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:20.457475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:20.457654 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 04:16:20.457762 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:16:20.457896 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260810 04:16:20.458765 19873 heartbeater.cc:347] Connected to a master server at 127.15.216.126:38281
I20260810 04:16:20.458884 19873 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:20.459067 19873 heartbeater.cc:511] Master 127.15.216.126:38281 requested a full tablet report, sending...
I20260810 04:16:20.459460 19517 ts_manager.cc:194] Registered new tserver with Master: 5058dae69ca646558fee8cc0e2c2bd4d (127.15.216.66:46221)
I20260810 04:16:20.459842 19517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:42275
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:20.484652 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38281
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:20.572798 19878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:20.573021 19878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:20.573047 19878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:20.574297 19878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:20.574345 19878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:20.574396 19878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:20.575898 19878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:20.576195 19878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:20.576483 19878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:20.582006 19884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.582008 19888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:20.582068 19885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:20.582254 19878 server_base.cc:1034] running on GCE node
I20260810 04:16:20.582518 19878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:20.582762 19878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:20.583981 19878 hybrid_clock.cc:648] HybridClock initialized: now 1786335380583954 us; error 35 us; skew 500 ppm
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:20.591068 19878 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:20.591626 19878 webserver.cc:460] Webserver started at http://127.15.216.67:43759/ using document root <none> and password file <none>
I20260810 04:16:20.591863 19878 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:20.591941 19878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:20.592099 19878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:20.593163 19878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "4dcbd7a9ada449819ba6b3fcebeb8f9d"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "85d2fc9fbb6ea623193b8bf915efe765"
server_key_iv: "2fe5d01a97dc281e94af586487de39f0"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.593536 19878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "4dcbd7a9ada449819ba6b3fcebeb8f9d"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "85d2fc9fbb6ea623193b8bf915efe765"
server_key_iv: "2fe5d01a97dc281e94af586487de39f0"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.594985 19878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:20.595846 19895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.596107 19878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:20.596258 19878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "4dcbd7a9ada449819ba6b3fcebeb8f9d"
format_stamp: "Formatted at 2026-08-10 04:16:20 on dist-test-slave-g350"
server_key: "85d2fc9fbb6ea623193b8bf915efe765"
server_key_iv: "2fe5d01a97dc281e94af586487de39f0"
server_key_version: "encryptionkey@0"
I20260810 04:16:20.596359 19878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:20.604681 19878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:20.607221 19878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:20.607393 19878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:20.607635 19878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:20.607985 19878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:20.608045 19878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.608094 19878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:20.608121 19878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:20.615324 19878 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:46573
I20260810 04:16:20.615381 20017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:46573 every 8 connection(s)
I20260810 04:16:20.615770 19878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:20.620991 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 19878
I20260810 04:16:20.621116 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:20.621362 16225 external_mini_cluster.cc:1428] Setting key aff8d6b591448c093311a1d33fc5cd4f
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:20.626358 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.616677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:46911 (local address 127.15.216.126:38281)
0810 04:16:20.616756 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:20.616759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:20.616956 (+ 197us) server_negotiation.cc:407] Connection header received
0810 04:16:20.617514 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:20.617516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:20.617529 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:20.617660 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:20.618685 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.619201 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.620783 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.620967 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.621024 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.623556 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:20.623557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:20.623557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:20.623564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:20.625049 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.625454 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.625454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.625455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.625504 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.625798 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.625798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.625799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.625901 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:20.625955 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:20.626193 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 04:16:20.626293 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:16:20.626497 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.616632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38281 (local address 127.15.216.67:46911)
0810 04:16:20.616824 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:20.616861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:20.617386 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:20.617751 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:20.617755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:20.618371 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:20.618572 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:20.618577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.619415 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.619417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:20.620697 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:20.620700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.621061 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.621063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:20.621234 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.621874 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:20.621882 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:20.623468 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:20.625137 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:20.625145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:20.625146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:20.625383 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:20.625562 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:20.625563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:20.625563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:20.625738 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:20.625962 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:20.625967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:20.626081 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 04:16:20.626165 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 04:16:20.626284 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":123,"threads_started":1}
I20260810 04:16:20.627003 20018 heartbeater.cc:347] Connected to a master server at 127.15.216.126:38281
I20260810 04:16:20.627099 20018 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:20.627250 20018 heartbeater.cc:511] Master 127.15.216.126:38281 requested a full tablet report, sending...
I20260810 04:16:20.627658 19517 ts_manager.cc:194] Registered new tserver with Master: 4dcbd7a9ada449819ba6b3fcebeb8f9d (127.15.216.67:46573)
I20260810 04:16:20.628096 19517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:46911
I20260810 04:16:20.633651 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[19467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335380, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:20.660100 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:20.651682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.15.216.126:38281)
0810 04:16:20.651789 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:20.651793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:20.651917 (+ 124us) server_negotiation.cc:407] Connection header received
0810 04:16:20.652025 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:20.652026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:20.652047 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:20.652124 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:20.652840 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.653408 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.654361 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:20.654577 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:20.654705 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:20.657211 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:20.657213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:20.657213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:20.657223 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:20.658751 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.659153 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.659154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.659155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.659220 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:20.659472 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:20.659473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:20.659473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:20.659646 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:20.659761 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:20.659897 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 04:16:20.659993 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 04:16:20.660386 19517 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57388
I20260810 04:16:20.661026 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 19588
I20260810 04:16:20.667732 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 19731
I20260810 04:16:20.674297 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 19878
I20260810 04:16:20.682732 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 19482
2026-08-10T04:16:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2004 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[20031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[20031](info): set up 2 sockets
Aug 10 04:16:20 dist-test-slave-g350 krb5kdc[20031](info): commencing operation
krb5kdc: starting...
W20260810 04:16:21.786208 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:21 dist-test-slave-g350 krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:21.921139 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41035
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:22.008901 20046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:22.009155 20046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:22.009196 20046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:22.010365 20046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:22.010464 20046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:22.010491 20046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:22.010524 20046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:22.010553 20046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:22.012071 20046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:22.012351 20046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:22.012588 20046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:22.017576 20053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:22.017637 20051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:22.017635 20057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:22.017793 20046 server_base.cc:1034] running on GCE node
I20260810 04:16:22.018118 20046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:22.018415 20046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:22.019563 20046 hybrid_clock.cc:648] HybridClock initialized: now 1786335382019544 us; error 36 us; skew 500 ppm
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:22.027072 20046 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:22.027539 20046 webserver.cc:460] Webserver started at http://127.15.216.126:39893/ using document root <none> and password file <none>
I20260810 04:16:22.027720 20046 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:22.027760 20046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:22.027849 20046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:22.028895 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "4e69246b6ba14af3b81ffb8cb99c942d"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "51bb89632f35bb08359e93a0ef9e34a4"
server_key_iv: "db2e9f1dbf25170b00aa758a85bb3140"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.029199 20046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "4e69246b6ba14af3b81ffb8cb99c942d"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "51bb89632f35bb08359e93a0ef9e34a4"
server_key_iv: "db2e9f1dbf25170b00aa758a85bb3140"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.030575 20046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:22.031288 20064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.031498 20046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:22.031610 20046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "4e69246b6ba14af3b81ffb8cb99c942d"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "51bb89632f35bb08359e93a0ef9e34a4"
server_key_iv: "db2e9f1dbf25170b00aa758a85bb3140"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.031692 20046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:22.046582 20046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:22.048919 20046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:22.049086 20046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:22.053488 20046 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41035
I20260810 04:16:22.053526 20121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41035 every 8 connection(s)
I20260810 04:16:22.054090 20046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:22.054766 20122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:22.057106 20122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d: Bootstrap starting.
I20260810 04:16:22.057168 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20046
I20260810 04:16:22.057281 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:22.057466 16225 external_mini_cluster.cc:1428] Setting key 7b91a349051f91221fb4b98ac5b41e8e
I20260810 04:16:22.057919 20122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:22.058275 20122 log.cc:824] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:22.059037 20122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d: No bootstrap required, opened a new log
I20260810 04:16:22.061323 20122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41035 } }
I20260810 04:16:22.061499 20122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:22.061528 20122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e69246b6ba14af3b81ffb8cb99c942d, State: Initialized, Role: FOLLOWER
I20260810 04:16:22.061691 20122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41035 } }
I20260810 04:16:22.061784 20122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:22.061810 20122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:22.061875 20122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:22.062471 20122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41035 } }
I20260810 04:16:22.062578 20122 leader_election.cc:302] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e69246b6ba14af3b81ffb8cb99c942d; no voters:
I20260810 04:16:22.062763 20122 leader_election.cc:288] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:22.062889 20127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:22.063066 20127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [term 1 LEADER]: Becoming Leader. State: Replica: 4e69246b6ba14af3b81ffb8cb99c942d, State: Running, Role: LEADER
I20260810 04:16:22.063103 20122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:22.063254 20127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41035 } }
I20260810 04:16:22.063602 20127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41035 } } }
I20260810 04:16:22.063680 20127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:22.063831 20127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e69246b6ba14af3b81ffb8cb99c942d. Latest consensus state: current_term: 1 leader_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69246b6ba14af3b81ffb8cb99c942d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41035 } } }
I20260810 04:16:22.063894 20127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:22.065277 20134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:22.065883 20134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:22.068892 20134 catalog_manager.cc:1347] Generated new cluster ID: 1d90b1a1832a4aa7a9e9cd34338ded14
I20260810 04:16:22.068945 20134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:22.070040 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.058057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.15.216.126:41035)
0810 04:16:22.058274 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:22.058282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:22.058316 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:16:22.058885 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:22.058894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:22.058953 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:22.059113 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:22.059866 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.061308 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.062286 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.062938 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.063082 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.066310 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:22.066315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:22.066315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:22.066326 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:22.068274 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.068728 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.068731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.068731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.068843 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.069086 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.069088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.069088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.069382 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:22.069530 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:22.069710 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 04:16:22.069838 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260810 04:16:22.084146 20134 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:22.084744 20134 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:22.089763 20134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e69246b6ba14af3b81ffb8cb99c942d: Generated new TSK 0
I20260810 04:16:22.089983 20134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:22.126497 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41035
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:22.211678 20149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:22.211886 20149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:22.211913 20149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:22.213057 20149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:22.213097 20149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:22.213124 20149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:22.214743 20149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:22.214936 20149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:22.215103 20149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:22.220242 20157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:22.220242 20154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:22.220397 20155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:22.220687 20149 server_base.cc:1034] running on GCE node
I20260810 04:16:22.220924 20149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:22.221161 20149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:22.222325 20149 hybrid_clock.cc:648] HybridClock initialized: now 1786335382222315 us; error 33 us; skew 500 ppm
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:22.225375 20149 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:22.225941 20149 webserver.cc:460] Webserver started at http://127.15.216.65:38151/ using document root <none> and password file <none>
I20260810 04:16:22.226227 20149 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:22.226306 20149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:22.226482 20149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:22.227608 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "5d579c44daf345239acde1235d9a87b8"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "32dffe83b4c035a5c1685569b3c750ee"
server_key_iv: "0c0d400ffebfab176a4d3405a0bd1360"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.228019 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "5d579c44daf345239acde1235d9a87b8"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "32dffe83b4c035a5c1685569b3c750ee"
server_key_iv: "0c0d400ffebfab176a4d3405a0bd1360"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.229692 20149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:22.230619 20166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.230886 20149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:22.230988 20149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "5d579c44daf345239acde1235d9a87b8"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "32dffe83b4c035a5c1685569b3c750ee"
server_key_iv: "0c0d400ffebfab176a4d3405a0bd1360"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.231082 20149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:22.238029 20149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:22.240011 20149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:22.240175 20149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:22.240414 20149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:22.240722 20149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:22.240778 20149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.240814 20149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:22.240861 20149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.247128 20149 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:42221
I20260810 04:16:22.247171 20287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:42221 every 8 connection(s)
I20260810 04:16:22.247550 20149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:22.252226 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20149
I20260810 04:16:22.252377 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:22.252589 16225 external_mini_cluster.cc:1428] Setting key 18f5d4a99eea1f8feb427f4399ed7ac4
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:22.258442 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.248363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:40973 (local address 127.15.216.126:41035)
0810 04:16:22.248420 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:22.248423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:22.248673 (+ 250us) server_negotiation.cc:407] Connection header received
0810 04:16:22.249784 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:22.249785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:22.249798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:22.249866 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:22.250632 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.251200 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.252418 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.252683 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.252745 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.255410 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:22.255411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:22.255412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:22.255421 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:22.256849 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.257363 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.257363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.257364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.257417 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.257719 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.257720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.257720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.257837 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:22.257894 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:22.258276 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 04:16:22.258375 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:16:22.258703 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.248347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41035 (local address 127.15.216.65:40973)
0810 04:16:22.248561 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:22.248601 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:22.249525 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:22.250002 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:22.250006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:22.250308 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:22.250525 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:22.250530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.251460 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.251461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:22.252283 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:22.252286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.252769 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.252769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:22.252920 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.253552 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:22.253561 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:22.255275 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:22.256928 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:22.256935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:22.256938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:22.257265 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:22.257483 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:22.257484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:22.257485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:22.257651 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:22.257920 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:22.257928 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:22.258108 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 04:16:22.258233 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 04:16:22.258426 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260810 04:16:22.259262 20288 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41035
I20260810 04:16:22.259405 20288 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:22.259620 20288 heartbeater.cc:511] Master 127.15.216.126:41035 requested a full tablet report, sending...
I20260810 04:16:22.260157 20083 ts_manager.cc:194] Registered new tserver with Master: 5d579c44daf345239acde1235d9a87b8 (127.15.216.65:42221)
I20260810 04:16:22.260763 20083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:40973
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:22.282903 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41035
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:22.367869 20293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:22.368103 20293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:22.368146 20293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:22.369284 20293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:22.369352 20293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:22.369406 20293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:22.370944 20293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:22.371224 20293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:22.371479 20293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:22.377116 20300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:22.377161 20299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:22.377149 20293 server_base.cc:1034] running on GCE node
W20260810 04:16:22.377116 20302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:22.377727 20293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:22.377987 20293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:22.379176 20293 hybrid_clock.cc:648] HybridClock initialized: now 1786335382379159 us; error 31 us; skew 500 ppm
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:22.382181 20293 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:22.382608 20293 webserver.cc:460] Webserver started at http://127.15.216.66:45677/ using document root <none> and password file <none>
I20260810 04:16:22.382822 20293 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:22.382864 20293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:22.382958 20293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:22.384007 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "b880793675474d08babefbbea4931cd8"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "fdb7a7457559ffbf6289a3dd1c626905"
server_key_iv: "57a96e3db0eb7696197f2d5d495484b5"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.384351 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "b880793675474d08babefbbea4931cd8"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "fdb7a7457559ffbf6289a3dd1c626905"
server_key_iv: "57a96e3db0eb7696197f2d5d495484b5"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.385828 20293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:16:22.387002 20309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.387214 20293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:22.387342 20293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "b880793675474d08babefbbea4931cd8"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "fdb7a7457559ffbf6289a3dd1c626905"
server_key_iv: "57a96e3db0eb7696197f2d5d495484b5"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.387455 20293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:22.401273 20293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:22.403635 20293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:22.403824 20293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:22.404103 20293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:22.404446 20293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:22.404505 20293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.404551 20293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:22.404598 20293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.411345 20293 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:41555
I20260810 04:16:22.411392 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:41555 every 8 connection(s)
I20260810 04:16:22.411777 20293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:22.418960 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20293
I20260810 04:16:22.419080 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:22.419273 16225 external_mini_cluster.cc:1428] Setting key d79d8d6f5f73d59548a389f73648432f
I20260810 04:16:22.422350 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.412840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:40487 (local address 127.15.216.126:41035)
0810 04:16:22.412934 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:22.412936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:22.413089 (+ 153us) server_negotiation.cc:407] Connection header received
0810 04:16:22.414054 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:22.414056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:22.414067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:22.414160 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:22.414986 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.415537 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.416599 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.416857 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.416916 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.419397 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:22.419399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:22.419399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:22.419408 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:22.420900 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.421329 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.421330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.421331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.421392 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.421689 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.421690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.421690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.421806 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:22.421883 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:22.422169 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 04:16:22.422262 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 04:16:22.422539 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.412713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41035 (local address 127.15.216.66:40487)
0810 04:16:22.412929 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:22.412968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:22.413945 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:22.414380 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:22.414384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:22.414678 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:22.414881 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:22.414885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.415634 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.415635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:22.416475 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:22.416477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.416960 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.416961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:22.417071 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.417777 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:22.417786 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:22.419288 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:22.421003 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:22.421012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:22.421014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:22.421247 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:22.421467 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:22.421468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:22.421469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:22.421586 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:22.421874 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:22.421882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:22.422027 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 04:16:22.422130 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:16:22.422294 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260810 04:16:22.423259 20432 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41035
I20260810 04:16:22.423367 20432 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:22.423599 20432 heartbeater.cc:511] Master 127.15.216.126:41035 requested a full tablet report, sending...
I20260810 04:16:22.423985 20083 ts_manager.cc:194] Registered new tserver with Master: b880793675474d08babefbbea4931cd8 (127.15.216.66:41555)
I20260810 04:16:22.424501 20083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:40487
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:22.449030 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41035
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:22.534622 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:22.534861 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:22.534904 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:22.536185 20437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:22.536268 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:22.536316 20437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:22.537966 20437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:22.538261 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:22.538506 20437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:22.544107 20445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:22.544143 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:22.544217 20437 server_base.cc:1034] running on GCE node
W20260810 04:16:22.544066 20442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:22.544503 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:22.544785 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:22.545933 20437 hybrid_clock.cc:648] HybridClock initialized: now 1786335382545895 us; error 54 us; skew 500 ppm
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:22.548556 20437 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:22.549032 20437 webserver.cc:460] Webserver started at http://127.15.216.67:41241/ using document root <none> and password file <none>
I20260810 04:16:22.549244 20437 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:22.549285 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:22.549384 20437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:22.550519 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "d46d7b79d9044e6e8affa0097dcf7b8b"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "7d4fd6f5ce71657fe5d2cf0e944c9838"
server_key_iv: "b63fc378621ccbea6bc191696beee7f9"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.550832 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "d46d7b79d9044e6e8affa0097dcf7b8b"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "7d4fd6f5ce71657fe5d2cf0e944c9838"
server_key_iv: "b63fc378621ccbea6bc191696beee7f9"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.552551 20437 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 04:16:22.553392 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.553578 20437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:22.553694 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "d46d7b79d9044e6e8affa0097dcf7b8b"
format_stamp: "Formatted at 2026-08-10 04:16:22 on dist-test-slave-g350"
server_key: "7d4fd6f5ce71657fe5d2cf0e944c9838"
server_key_iv: "b63fc378621ccbea6bc191696beee7f9"
server_key_version: "encryptionkey@0"
I20260810 04:16:22.553752 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:22.560212 20437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:22.562570 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:22.562683 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:22.562879 20437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:22.563202 20437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:22.563238 20437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.563259 20437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:22.563308 20437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:22.569484 20437 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:37315
I20260810 04:16:22.569574 20572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:37315 every 8 connection(s)
I20260810 04:16:22.569929 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:22.575287 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20437
I20260810 04:16:22.575392 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:22.575598 16225 external_mini_cluster.cc:1428] Setting key 5765fcdfe45b4f55cff8e524be66b212
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:22.580018 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.570767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:39171 (local address 127.15.216.126:41035)
0810 04:16:22.570860 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:22.570862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:22.571037 (+ 175us) server_negotiation.cc:407] Connection header received
0810 04:16:22.572039 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:22.572040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:22.572054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:22.572106 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:22.572752 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.573257 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.574366 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.574577 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.574653 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.577248 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:22.577249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:22.577250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:22.577258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:22.578589 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.579011 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.579013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.579013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.579073 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.579356 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.579357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.579358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.579474 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:22.579550 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:22.579843 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 04:16:22.579928 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 04:16:22.580217 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.570683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41035 (local address 127.15.216.67:39171)
0810 04:16:22.570899 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:22.570947 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:22.571913 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:22.572173 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:22.572182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:22.572446 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:22.572645 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:22.572649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.573362 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.573364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:22.574268 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:22.574271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.574665 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.574666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:22.574805 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.575566 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:22.575574 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:22.577124 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:22.578693 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:22.578701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:22.578703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:22.578930 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:22.579155 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:22.579156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:22.579156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:22.579270 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:22.579542 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:22.579549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:22.579700 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 04:16:22.579809 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:16:22.579971 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260810 04:16:22.580801 20574 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41035
I20260810 04:16:22.580925 20574 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:22.581110 20574 heartbeater.cc:511] Master 127.15.216.126:41035 requested a full tablet report, sending...
I20260810 04:16:22.581578 20083 ts_manager.cc:194] Registered new tserver with Master: d46d7b79d9044e6e8affa0097dcf7b8b (127.15.216.67:37315)
I20260810 04:16:22.582000 20083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:39171
I20260810 04:16:22.587692 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:22.611982 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.604020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.15.216.65:42221)
0810 04:16:22.604195 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:22.604201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:22.604212 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:22.604240 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:22.604241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:22.604256 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:22.604324 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:22.604901 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.605750 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.606490 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.606833 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.606928 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.609203 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:22.609211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:22.609211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:22.609218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:22.610703 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.611095 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.611098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.611098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.611168 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.611381 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.611382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.611382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.611525 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:22.611680 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:22.611766 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:16:22.611856 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:22.620457 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:22.613268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.15.216.126:41035)
0810 04:16:22.613401 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:22.613404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:22.613412 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:22.613437 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:22.613438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:22.613448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:22.613501 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:22.614040 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.614529 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.615212 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:22.615424 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:22.615507 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:22.617789 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:22.617790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:22.617791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:22.617798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:22.619124 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.619610 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.619611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.619612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.619673 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:22.619880 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:22.619881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:22.619882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:22.620043 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:22.620157 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:22.620269 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 04:16:22.620351 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260810 04:16:22.620724 20083 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38150
I20260810 04:16:22.621098 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20149
I20260810 04:16:22.632365 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20293
I20260810 04:16:22.638484 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20437
I20260810 04:16:22.644085 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20046
2026-08-10T04:16:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1960 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20587](info): set up 2 sockets
Aug 10 04:16:22 dist-test-slave-g350 krb5kdc[20587](info): commencing operation
krb5kdc: starting...
W20260810 04:16:23.742007 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:23 dist-test-slave-g350 krb5kdc[20587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:23.878229 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:42497
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:42497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:23.964685 20601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:23.964942 20601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:23.964984 20601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:23.966249 20601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:23.966315 20601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:23.966339 20601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:23.966367 20601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:23.966398 20601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:23.967921 20601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:42497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:42497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:23.968210 20601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:23.968446 20601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:23.973826 20606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:23.973826 20607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:23.973855 20601 server_base.cc:1034] running on GCE node
W20260810 04:16:23.973855 20609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:23.974318 20601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:23.974640 20601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:23.975826 20601 hybrid_clock.cc:648] HybridClock initialized: now 1786335383975827 us; error 46 us; skew 500 ppm
Aug 10 04:16:23 dist-test-slave-g350 krb5kdc[20587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335383, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:23.978614 20601 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:23.978999 20601 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:23.979074 20601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:23.979233 20601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:23.980443 20601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "88fe812a8c5f4347919aa32598e38b31"
format_stamp: "Formatted at 2026-08-10 04:16:23 on dist-test-slave-g350"
server_key: "590ef4a245a3352b2a90faceb2f279bb"
server_key_iv: "7f0417283c7bc431ce740dd950998e20"
server_key_version: "encryptionkey@0"
I20260810 04:16:23.980854 20601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "88fe812a8c5f4347919aa32598e38b31"
format_stamp: "Formatted at 2026-08-10 04:16:23 on dist-test-slave-g350"
server_key: "590ef4a245a3352b2a90faceb2f279bb"
server_key_iv: "7f0417283c7bc431ce740dd950998e20"
server_key_version: "encryptionkey@0"
I20260810 04:16:23.982391 20601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:23.983251 20616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:23.983475 20601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:23.983592 20601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "88fe812a8c5f4347919aa32598e38b31"
format_stamp: "Formatted at 2026-08-10 04:16:23 on dist-test-slave-g350"
server_key: "590ef4a245a3352b2a90faceb2f279bb"
server_key_iv: "7f0417283c7bc431ce740dd950998e20"
server_key_version: "encryptionkey@0"
I20260810 04:16:23.983667 20601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:24.004715 20601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:24.007085 20601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:24.007278 20601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:24.011783 20601 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:42497
I20260810 04:16:24.011809 20669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:42497 every 8 connection(s)
I20260810 04:16:24.012125 20601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:24.013145 20670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:24.014289 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20601
I20260810 04:16:24.014420 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:24.014604 16225 external_mini_cluster.cc:1428] Setting key 7324de886f891f0100bad0e498d85391
I20260810 04:16:24.016124 20670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31: Bootstrap starting.
I20260810 04:16:24.016764 20670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:24.017093 20670 log.cc:824] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:24.017868 20670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31: No bootstrap required, opened a new log
I20260810 04:16:24.019550 20670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe812a8c5f4347919aa32598e38b31" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42497 } }
I20260810 04:16:24.019735 20670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:24.019775 20670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88fe812a8c5f4347919aa32598e38b31, State: Initialized, Role: FOLLOWER
I20260810 04:16:24.019914 20670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe812a8c5f4347919aa32598e38b31" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42497 } }
I20260810 04:16:24.020015 20670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:24.020043 20670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:24.020098 20670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:24.020638 20670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe812a8c5f4347919aa32598e38b31" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42497 } }
I20260810 04:16:24.020730 20670 leader_election.cc:302] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88fe812a8c5f4347919aa32598e38b31; no voters:
I20260810 04:16:24.020952 20670 leader_election.cc:288] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:24.020988 20675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:24.021093 20675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [term 1 LEADER]: Becoming Leader. State: Replica: 88fe812a8c5f4347919aa32598e38b31, State: Running, Role: LEADER
I20260810 04:16:24.021186 20675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe812a8c5f4347919aa32598e38b31" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42497 } }
I20260810 04:16:24.021221 20670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:24.021526 20675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88fe812a8c5f4347919aa32598e38b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe812a8c5f4347919aa32598e38b31" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42497 } } }
I20260810 04:16:24.021620 20675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:24.021766 20675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88fe812a8c5f4347919aa32598e38b31. Latest consensus state: current_term: 1 leader_uuid: "88fe812a8c5f4347919aa32598e38b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe812a8c5f4347919aa32598e38b31" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42497 } } }
I20260810 04:16:24.021848 20675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31 [sys.catalog]: This master's current role is: LEADER
Aug 10 04:16:24 dist-test-slave-g350 krb5kdc[20587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:24.022087 20680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:24.022841 20680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:24.024310 20680 catalog_manager.cc:1347] Generated new cluster ID: 3cf0e328bded459fb3e0b7d5c5e02752
I20260810 04:16:24.024380 20680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:24.026340 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.015203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.15.216.126:42497)
0810 04:16:24.015411 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.015420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.015454 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:16:24.015995 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.016005 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.016066 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.016238 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:24.016999 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.018115 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.018870 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.019289 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.019428 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.022503 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:24.022507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:24.022507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:24.022519 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:24.024624 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.025034 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.025038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.025038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.025148 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.025361 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.025362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.025363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.025707 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:24.025844 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.025965 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.026109 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260810 04:16:24.045418 20680 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:24.046042 20680 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:24.053936 20680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31: Generated new TSK 0
I20260810 04:16:24.054147 20680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:24.093714 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:42497
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:24.179399 20696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:24.179682 20696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:24.179730 20696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:24.180966 20696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:24.181051 20696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:24.181102 20696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:24.182940 20696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:24.183250 20696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:24.183524 20696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:24.189651 20705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.189674 20702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:24.189766 20696 server_base.cc:1034] running on GCE node
W20260810 04:16:24.189786 20703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:24.190218 20696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:24.190502 20696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:24.191700 20696 hybrid_clock.cc:648] HybridClock initialized: now 1786335384191646 us; error 75 us; skew 500 ppm
Aug 10 04:16:24 dist-test-slave-g350 krb5kdc[20587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:24.194676 20696 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:24.195089 20696 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:24.195169 20696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:24.195338 20696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:24.196529 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "04eb8c1e23f6471d9d660f5fb416b623"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "00db85e366f18694c0d2966d2ac5a4e0"
server_key_iv: "dfb0ff3e7007c2a707b23b96ea112043"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.196915 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "04eb8c1e23f6471d9d660f5fb416b623"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "00db85e366f18694c0d2966d2ac5a4e0"
server_key_iv: "dfb0ff3e7007c2a707b23b96ea112043"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.198462 20696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:24.199363 20712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.199579 20696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:24.199718 20696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "04eb8c1e23f6471d9d660f5fb416b623"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "00db85e366f18694c0d2966d2ac5a4e0"
server_key_iv: "dfb0ff3e7007c2a707b23b96ea112043"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.199803 20696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:24.210210 20696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:24.212796 20696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:24.212983 20696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:24.213258 20696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:24.213655 20696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:24.213716 20696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.213758 20696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:24.213788 20696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.220229 20696 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:39829
I20260810 04:16:24.220326 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:39829 every 8 connection(s)
I20260810 04:16:24.220619 20696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:16:24 dist-test-slave-g350 krb5kdc[20587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:24.230515 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20696
I20260810 04:16:24.230657 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:24.230906 16225 external_mini_cluster.cc:1428] Setting key 2af1afc94cdbacbeeaf8bc4700ef8eca
I20260810 04:16:24.231199 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.221473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:58207 (local address 127.15.216.126:42497)
0810 04:16:24.221532 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.221534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.221872 (+ 338us) server_negotiation.cc:407] Connection header received
0810 04:16:24.222370 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.222372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.222381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.222440 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:24.223495 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.224021 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.225023 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.225222 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.225289 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.227631 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:24.227633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:24.227633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:24.227642 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:24.229167 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.229645 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.229646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.229646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.229710 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.230116 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.230117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.230117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.230414 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:24.230520 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.230928 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.231050 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:16:24.231428 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.221448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:42497 (local address 127.15.216.65:58207)
0810 04:16:24.221676 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.221768 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.222283 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.222515 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.222518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.222799 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:24.223392 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.223398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.224111 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.224112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.224934 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.224936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.225300 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.225301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.225422 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.226046 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:24.226059 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:24.227519 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:24.229261 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:24.229268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:24.229270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:24.229518 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:24.229849 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:24.229850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:24.229851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:24.229991 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:24.230522 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:24.230529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:24.230750 (+ 221us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.230874 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.231109 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
I20260810 04:16:24.231969 20833 heartbeater.cc:347] Connected to a master server at 127.15.216.126:42497
I20260810 04:16:24.232100 20833 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:24.232306 20833 heartbeater.cc:511] Master 127.15.216.126:42497 requested a full tablet report, sending...
I20260810 04:16:24.232969 20633 ts_manager.cc:194] Registered new tserver with Master: 04eb8c1e23f6471d9d660f5fb416b623 (127.15.216.65:39829)
I20260810 04:16:24.233891 20633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:58207
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:24.262692 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:42497
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:24.354336 20838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:24.354601 20838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:24.354645 20838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:24.355870 20838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:24.355945 20838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:24.356016 20838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:24.357712 20838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:24.357992 20838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:24.358253 20838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:24.363636 20847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.363601 20844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.363626 20845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:24.363934 20838 server_base.cc:1034] running on GCE node
I20260810 04:16:24.364187 20838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:24.364441 20838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:24.365654 20838 hybrid_clock.cc:648] HybridClock initialized: now 1786335384365631 us; error 31 us; skew 500 ppm
Aug 10 04:16:24 dist-test-slave-g350 krb5kdc[20587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:24.368484 20838 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:24.368870 20838 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:24.368944 20838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:24.369102 20838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:24.370276 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "5810da3663d715f8aab9704a67b2457b"
server_key_iv: "e6f41334b6aaf4559fb13e56c9d24b9f"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.370652 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "5810da3663d715f8aab9704a67b2457b"
server_key_iv: "e6f41334b6aaf4559fb13e56c9d24b9f"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.372092 20838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:16:24.373014 20854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.373231 20838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:24.373344 20838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "5810da3663d715f8aab9704a67b2457b"
server_key_iv: "e6f41334b6aaf4559fb13e56c9d24b9f"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.373440 20838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:24.384187 20838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:24.386754 20838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:24.386966 20838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:24.387269 20838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:24.387621 20838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:24.387678 20838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.387723 20838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:24.387751 20838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.394685 20838 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:37591
I20260810 04:16:24.394718 20977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:37591 every 8 connection(s)
I20260810 04:16:24.395030 20838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:24.399082 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20838
I20260810 04:16:24.399215 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:24.399477 16225 external_mini_cluster.cc:1428] Setting key 723af01c49fd3fd280935a604d986f51
Aug 10 04:16:24 dist-test-slave-g350 krb5kdc[20587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:24.406150 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.395802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:36011 (local address 127.15.216.126:42497)
0810 04:16:24.395887 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.395890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.396097 (+ 207us) server_negotiation.cc:407] Connection header received
0810 04:16:24.396593 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.396594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.396606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.396669 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:24.397719 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.398219 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.399445 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.399781 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.399895 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.402469 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:24.402471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:24.402471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:24.402481 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:24.404463 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.404922 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.404924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.404924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.404996 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.405289 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.405290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.405291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.405462 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:24.405542 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.405910 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.406006 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 04:16:24.406183 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.395751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:42497 (local address 127.15.216.66:36011)
0810 04:16:24.395974 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.396011 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.396487 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.396743 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.396746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.397041 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:24.397562 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.397567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.398321 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.398321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.399353 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.399356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.399913 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.399915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.400066 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.400702 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:24.400712 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:24.402362 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:24.404558 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:24.404566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:24.404568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:24.404834 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:24.405079 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:24.405079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:24.405080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:24.405206 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:24.405546 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:24.405552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:24.405753 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.405855 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.405999 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":116,"threads_started":1}
I20260810 04:16:24.406889 20978 heartbeater.cc:347] Connected to a master server at 127.15.216.126:42497
I20260810 04:16:24.407032 20978 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:24.407200 20978 heartbeater.cc:511] Master 127.15.216.126:42497 requested a full tablet report, sending...
I20260810 04:16:24.407665 20633 ts_manager.cc:194] Registered new tserver with Master: 4b1e04566b4a450c8ca11d4b5ae3cc60 (127.15.216.66:37591)
I20260810 04:16:24.408205 20633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:36011
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:24.429445 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:42497
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:24.516198 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:24.516400 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:24.516425 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:24.517635 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:24.517702 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:24.517750 20984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:24.519290 20984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:24.519575 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:24.519812 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:24.525312 20993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.525476 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:24.525692 20984 server_base.cc:1034] running on GCE node
W20260810 04:16:24.525828 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:24.526117 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:24.526387 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:24.527587 20984 hybrid_clock.cc:648] HybridClock initialized: now 1786335384527574 us; error 28 us; skew 500 ppm
Aug 10 04:16:24 dist-test-slave-g350 krb5kdc[20587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:24.530483 20984 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:24.530843 20984 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:24.530889 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:24.530980 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:24.532060 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "912cebe95721435cbc00dfaeca7f9d75"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "bd159e33e3dda27c9f4886b8339c60fa"
server_key_iv: "a7f4747cf38e77a4b2821cf7b20e4131"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.532389 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "912cebe95721435cbc00dfaeca7f9d75"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "bd159e33e3dda27c9f4886b8339c60fa"
server_key_iv: "a7f4747cf38e77a4b2821cf7b20e4131"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.533779 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:24.534711 21000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.534952 20984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:24.535053 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "912cebe95721435cbc00dfaeca7f9d75"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "bd159e33e3dda27c9f4886b8339c60fa"
server_key_iv: "a7f4747cf38e77a4b2821cf7b20e4131"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.535126 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:24.546662 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:24.548934 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:24.549113 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:24.549379 20984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:24.549775 20984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:24.549834 20984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.549877 20984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:24.549904 20984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.556710 20984 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:43127
I20260810 04:16:24.556782 21123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:43127 every 8 connection(s)
I20260810 04:16:24.557096 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:16:24 dist-test-slave-g350 krb5kdc[20587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335384, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:24.565543 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 20984
I20260810 04:16:24.565681 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:24.565874 16225 external_mini_cluster.cc:1428] Setting key 973fb419c9f78856b562ac9219b64ad0
I20260810 04:16:24.567174 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.557936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:59195 (local address 127.15.216.126:42497)
0810 04:16:24.558012 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.558014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.558287 (+ 273us) server_negotiation.cc:407] Connection header received
0810 04:16:24.558695 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.558697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.558708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.558773 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:24.559686 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.560423 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.561450 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.561713 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.561771 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.564104 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:24.564105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:24.564106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:24.564112 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:24.565397 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.566000 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.566002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.566003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.566075 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.566451 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.566453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.566453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.566587 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:24.566669 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.567002 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.567082 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:16:24.567381 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.557907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:42497 (local address 127.15.216.67:59195)
0810 04:16:24.558069 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.558112 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.558598 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.558828 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.558831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.559099 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:24.559595 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.559600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.560521 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.560522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.561369 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.561371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.561800 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.561801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.561908 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.562517 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:24.562525 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:24.564000 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:24.565519 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:24.565527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:24.565529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:24.565906 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:24.566157 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:24.566158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:24.566158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:24.566368 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:24.566660 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:24.566667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:24.566855 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.566980 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.567140 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260810 04:16:24.567835 21124 heartbeater.cc:347] Connected to a master server at 127.15.216.126:42497
I20260810 04:16:24.567934 21124 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:24.568048 21124 heartbeater.cc:511] Master 127.15.216.126:42497 requested a full tablet report, sending...
I20260810 04:16:24.568416 20633 ts_manager.cc:194] Registered new tserver with Master: 912cebe95721435cbc00dfaeca7f9d75 (127.15.216.67:43127)
I20260810 04:16:24.568749 20633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:59195
I20260810 04:16:24.577879 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:16:24.585347 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.579208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.15.216.126:42497)
0810 04:16:24.579301 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.579303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.579380 (+ 77us) server_negotiation.cc:407] Connection header received
0810 04:16:24.579428 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.579429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.579438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.579488 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:24.580045 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.580551 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.581333 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.581523 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.581656 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.582946 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:24.582947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:24.582947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:24.582954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:24.584267 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.584616 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.584617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.584617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.584672 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:24.584854 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:24.584855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:24.584855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:24.585003 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:24.585083 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.585203 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.585276 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 04:16:24.587414 20633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:24.589308 20633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:24.596222 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.592012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.15.216.66:37591)
0810 04:16:24.592212 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.592217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.592229 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:16:24.592430 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.592434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.592451 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.592535 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:24.592782 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.593958 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595219 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.595719 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595825 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.595891 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.596066 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.596114 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260810 04:16:24.596231 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.591846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:39829 (local address 127.0.0.1:52524)
0810 04:16:24.592033 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.592076 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.592143 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.592555 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.592559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.592562 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:24.592637 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.592639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.594100 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.594101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.595017 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.595020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.595901 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.595991 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.595995 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.596027 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.596121 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260810 04:16:24.596427 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.592315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.15.216.67:43127)
0810 04:16:24.592510 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.592515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.592526 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:24.592563 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.592565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.592580 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.592653 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:24.592790 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.593558 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595022 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.595767 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595860 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.595948 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.596279 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.596332 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260810 04:16:24.596246 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.591879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:37591 (local address 127.0.0.1:44962)
0810 04:16:24.592171 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.592182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.592235 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.592630 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.592631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.592632 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:24.592700 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.592701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.594088 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.594089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.595129 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.595131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.595828 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.595939 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.595954 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.596049 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.596121 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":36,"thread_start_us":105,"threads_started":1}
I20260810 04:16:24.596236 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.592099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:43127 (local address 127.0.0.1:41262)
0810 04:16:24.592320 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.592329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.592371 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.592654 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.592655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.592657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:24.592730 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.592732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.593693 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.593694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.594932 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.594935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.595980 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.596054 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.596056 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.596105 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.596139 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":70,"threads_started":1}
I20260810 04:16:24.596997 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.591989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.15.216.65:39829)
0810 04:16:24.592328 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.592333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.592343 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:24.592392 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.592394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.592412 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.592498 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:24.592859 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.593963 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.595114 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.595805 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.596769 (+ 964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.596839 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.596868 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.596914 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":79,"threads_started":1}
I20260810 04:16:24.597278 20905 tablet_service.cc:1467] Processing CreateTablet for tablet 2432467627a648f1926f1ec1d5908659 (DEFAULT_TABLE table=test-table [id=401c3d4eeb51488992efd7d9842fb8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:24.597265 21050 tablet_service.cc:1467] Processing CreateTablet for tablet 2432467627a648f1926f1ec1d5908659 (DEFAULT_TABLE table=test-table [id=401c3d4eeb51488992efd7d9842fb8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:24.597688 21050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2432467627a648f1926f1ec1d5908659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:24.597688 20905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2432467627a648f1926f1ec1d5908659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:24.597581 20761 tablet_service.cc:1467] Processing CreateTablet for tablet 2432467627a648f1926f1ec1d5908659 (DEFAULT_TABLE table=test-table [id=401c3d4eeb51488992efd7d9842fb8ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:24.597911 20761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2432467627a648f1926f1ec1d5908659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:24.600749 21142 tablet_bootstrap.cc:492] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Bootstrap starting.
I20260810 04:16:24.601289 21143 tablet_bootstrap.cc:492] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623: Bootstrap starting.
I20260810 04:16:24.601655 21142 tablet_bootstrap.cc:654] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:24.601832 21144 tablet_bootstrap.cc:492] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Bootstrap starting.
I20260810 04:16:24.602013 21143 tablet_bootstrap.cc:654] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:24.602360 21142 log.cc:824] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:24.602605 21144 tablet_bootstrap.cc:654] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:24.602708 21143 log.cc:824] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:24.603219 21144 log.cc:824] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:24.603302 21142 tablet_bootstrap.cc:492] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: No bootstrap required, opened a new log
I20260810 04:16:24.603415 21142 ts_tablet_manager.cc:1397] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 04:16:24.604104 21144 tablet_bootstrap.cc:492] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: No bootstrap required, opened a new log
I20260810 04:16:24.604182 21144 ts_tablet_manager.cc:1397] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:16:24.605685 21142 raft_consensus.cc:348] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.605875 21142 raft_consensus.cc:374] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:24.605921 21142 raft_consensus.cc:729] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1e04566b4a450c8ca11d4b5ae3cc60, State: Initialized, Role: FOLLOWER
I20260810 04:16:24.606066 21142 consensus_queue.cc:260] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.606259 21143 tablet_bootstrap.cc:492] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623: No bootstrap required, opened a new log
I20260810 04:16:24.606341 21143 ts_tablet_manager.cc:1397] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260810 04:16:24.606389 21142 ts_tablet_manager.cc:1428] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 04:16:24.606324 21144 raft_consensus.cc:348] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.606487 21144 raft_consensus.cc:374] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:24.606528 21144 raft_consensus.cc:729] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912cebe95721435cbc00dfaeca7f9d75, State: Initialized, Role: FOLLOWER
I20260810 04:16:24.606541 20978 heartbeater.cc:503] Master 127.15.216.126:42497 was elected leader, sending a full tablet report...
I20260810 04:16:24.606631 21144 consensus_queue.cc:260] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.606932 21144 ts_tablet_manager.cc:1428] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:24.607071 21124 heartbeater.cc:503] Master 127.15.216.126:42497 was elected leader, sending a full tablet report...
I20260810 04:16:24.608619 21143 raft_consensus.cc:348] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.608749 21143 raft_consensus.cc:374] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:24.608771 21143 raft_consensus.cc:729] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04eb8c1e23f6471d9d660f5fb416b623, State: Initialized, Role: FOLLOWER
I20260810 04:16:24.608879 21143 consensus_queue.cc:260] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.609113 21143 ts_tablet_manager.cc:1428] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:16:24.609163 20833 heartbeater.cc:503] Master 127.15.216.126:42497 was elected leader, sending a full tablet report...
I20260810 04:16:24.620985 21150 raft_consensus.cc:482] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:24.621111 21150 raft_consensus.cc:504] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.621634 21150 leader_election.cc:288] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04eb8c1e23f6471d9d660f5fb416b623 (127.15.216.65:39829), 4b1e04566b4a450c8ca11d4b5ae3cc60 (127.15.216.66:37591)
I20260810 04:16:24.625686 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.621940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:42145 (local address 127.15.216.66:37591)
0810 04:16:24.621979 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.621981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.622093 (+ 112us) server_negotiation.cc:407] Connection header received
0810 04:16:24.622128 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.622129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.622139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.622197 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:24.622347 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.623080 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.624650 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.625253 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.625328 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.625337 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.625553 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.625614 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:16:24.625686 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.621884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:59313 (local address 127.15.216.65:39829)
0810 04:16:24.621945 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.621947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.621956 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:24.621985 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.621986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.622013 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.622077 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:24.622278 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.623097 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.624653 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.625247 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.625328 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.625337 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.625556 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.625617 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:16:24.625705 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.621770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:39829 (local address 127.15.216.67:59313)
0810 04:16:24.621820 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.621833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.621887 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.622084 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.622085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.622087 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:24.622199 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.622201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.623176 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.623177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.624545 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.624548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.625379 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.625379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.625460 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.625468 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.625531 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.625625 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":72}
I20260810 04:16:24.625717 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.621803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:37591 (local address 127.15.216.67:42145)
0810 04:16:24.622015 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.622024 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:24.622077 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:24.622211 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:24.622212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:24.622214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:24.622278 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.622279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.623176 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.623177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.624545 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:24.624548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.625366 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.625367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:24.625460 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.625468 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 04:16:24.625531 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:16:24.625613 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":101,"thread_start_us":130,"threads_started":1}
I20260810 04:16:24.626119 20783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2432467627a648f1926f1ec1d5908659" candidate_uuid: "912cebe95721435cbc00dfaeca7f9d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04eb8c1e23f6471d9d660f5fb416b623" is_pre_election: true
I20260810 04:16:24.626361 20783 raft_consensus.cc:2454] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912cebe95721435cbc00dfaeca7f9d75 in term 0.
I20260810 04:16:24.626672 21003 leader_election.cc:302] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04eb8c1e23f6471d9d660f5fb416b623, 912cebe95721435cbc00dfaeca7f9d75; no voters:
I20260810 04:16:24.626837 21150 raft_consensus.cc:2781] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:24.626814 20928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2432467627a648f1926f1ec1d5908659" candidate_uuid: "912cebe95721435cbc00dfaeca7f9d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" is_pre_election: true
I20260810 04:16:24.626905 21150 raft_consensus.cc:482] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:24.626955 21150 raft_consensus.cc:3037] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:24.626955 20928 raft_consensus.cc:2454] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912cebe95721435cbc00dfaeca7f9d75 in term 0.
I20260810 04:16:24.627836 21150 raft_consensus.cc:504] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.628006 21150 leader_election.cc:288] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [CANDIDATE]: Term 1 election: Requested vote from peers 04eb8c1e23f6471d9d660f5fb416b623 (127.15.216.65:39829), 4b1e04566b4a450c8ca11d4b5ae3cc60 (127.15.216.66:37591)
I20260810 04:16:24.628134 20783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2432467627a648f1926f1ec1d5908659" candidate_uuid: "912cebe95721435cbc00dfaeca7f9d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04eb8c1e23f6471d9d660f5fb416b623"
I20260810 04:16:24.628217 20783 raft_consensus.cc:3037] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:24.628204 20928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2432467627a648f1926f1ec1d5908659" candidate_uuid: "912cebe95721435cbc00dfaeca7f9d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60"
I20260810 04:16:24.628345 20928 raft_consensus.cc:3037] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:24.629254 20783 raft_consensus.cc:2454] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912cebe95721435cbc00dfaeca7f9d75 in term 1.
I20260810 04:16:24.629257 20928 raft_consensus.cc:2454] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912cebe95721435cbc00dfaeca7f9d75 in term 1.
I20260810 04:16:24.629492 21001 leader_election.cc:302] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b1e04566b4a450c8ca11d4b5ae3cc60, 912cebe95721435cbc00dfaeca7f9d75; no voters:
I20260810 04:16:24.629597 21150 raft_consensus.cc:2781] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:24.629789 21150 raft_consensus.cc:686] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 1 LEADER]: Becoming Leader. State: Replica: 912cebe95721435cbc00dfaeca7f9d75, State: Running, Role: LEADER
I20260810 04:16:24.629910 21150 consensus_queue.cc:237] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } }
I20260810 04:16:24.630905 20633 catalog_manager.cc:5495] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 912cebe95721435cbc00dfaeca7f9d75 (127.15.216.67). New cstate: current_term: 1 leader_uuid: "912cebe95721435cbc00dfaeca7f9d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912cebe95721435cbc00dfaeca7f9d75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43127 } health_report { overall_health: HEALTHY } } }
I20260810 04:16:24.642458 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.639884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.15.216.67:43127)
0810 04:16:24.639945 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.639948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.639957 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:24.640030 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.640031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.640052 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.640136 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:24.640306 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.640786 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.641621 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:24.641823 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:24.641906 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:24.642103 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:24.642200 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:24.642304 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.642323 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.642343 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:16:24.644558 20928 raft_consensus.cc:1261] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 1 FOLLOWER]: Refusing update from remote peer 912cebe95721435cbc00dfaeca7f9d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:24.644558 20783 raft_consensus.cc:1261] T 2432467627a648f1926f1ec1d5908659 P 04eb8c1e23f6471d9d660f5fb416b623 [term 1 FOLLOWER]: Refusing update from remote peer 912cebe95721435cbc00dfaeca7f9d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:24.645048 21150 consensus_queue.cc:1035] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b1e04566b4a450c8ca11d4b5ae3cc60" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:24.645221 21154 consensus_queue.cc:1035] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04eb8c1e23f6471d9d660f5fb416b623" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 39829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:16:24.645673 20979 tablet.cc:2369] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:24.648028 21157 mvcc.cc:205] Tried to move back new op lower bound from 7316829735500541952 to 7316829735444955136. Current Snapshot: MvccSnapshot[applied={T|T < 7316829735500541952}]
I20260810 04:16:24.649088 21158 mvcc.cc:205] Tried to move back new op lower bound from 7316829735500541952 to 7316829735444955136. Current Snapshot: MvccSnapshot[applied={T|T < 7316829735500541952 or (T in {7316829735500541952})}]
I20260810 04:16:24.650921 20632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:24.651046 20632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:24.651989 20632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88fe812a8c5f4347919aa32598e38b31: Sending DeleteTablet for 3 replicas of tablet 2432467627a648f1926f1ec1d5908659
I20260810 04:16:24.652369 20761 tablet_service.cc:1514] Processing DeleteTablet for tablet 2432467627a648f1926f1ec1d5908659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:24 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:52524
I20260810 04:16:24.652521 20905 tablet_service.cc:1514] Processing DeleteTablet for tablet 2432467627a648f1926f1ec1d5908659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:24 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:44962
I20260810 04:16:24.652614 21050 tablet_service.cc:1514] Processing DeleteTablet for tablet 2432467627a648f1926f1ec1d5908659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:24 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:41262
I20260810 04:16:24.652910 21168 tablet_replica.cc:330] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: stopping tablet replica
I20260810 04:16:24.653039 21168 raft_consensus.cc:2229] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:24.653159 21168 raft_consensus.cc:2258] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:24.653232 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20696
I20260810 04:16:24.655925 21168 ts_tablet_manager.cc:1905] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:24.657748 21168 ts_tablet_manager.cc:1918] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:24.657837 21168 log.cc:1196] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/2432467627a648f1926f1ec1d5908659
I20260810 04:16:24.658165 21168 ts_tablet_manager.cc:1939] T 2432467627a648f1926f1ec1d5908659 P 4b1e04566b4a450c8ca11d4b5ae3cc60: Deleting consensus metadata
I20260810 04:16:24.658816 20617 catalog_manager.cc:4841] TS 4b1e04566b4a450c8ca11d4b5ae3cc60 (127.15.216.66:37591): tablet 2432467627a648f1926f1ec1d5908659 (table test-table [id=401c3d4eeb51488992efd7d9842fb8ab]) successfully deleted
I20260810 04:16:24.659605 21169 tablet_replica.cc:330] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: stopping tablet replica
I20260810 04:16:24.659739 21169 raft_consensus.cc:2229] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:24.659864 21169 raft_consensus.cc:2258] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:24.660316 21169 ts_tablet_manager.cc:1905] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:24.662554 21169 ts_tablet_manager.cc:1918] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:24.662643 21169 log.cc:1196] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/2432467627a648f1926f1ec1d5908659
I20260810 04:16:24.662940 21169 ts_tablet_manager.cc:1939] T 2432467627a648f1926f1ec1d5908659 P 912cebe95721435cbc00dfaeca7f9d75: Deleting consensus metadata
I20260810 04:16:24.663656 20617 catalog_manager.cc:4841] TS 912cebe95721435cbc00dfaeca7f9d75 (127.15.216.67:43127): tablet 2432467627a648f1926f1ec1d5908659 (table test-table [id=401c3d4eeb51488992efd7d9842fb8ab]) successfully deleted
W20260810 04:16:24.665485 20619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:39829 (error 108)
I20260810 04:16:24.665766 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20838
I20260810 04:16:24.666153 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.665963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:39829 (local address 127.0.0.1:52532)
0810 04:16:24.666061 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:24.666092 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 04:16:24.666313 20619 catalog_manager.cc:4584] TS 04eb8c1e23f6471d9d660f5fb416b623 (127.15.216.65:39829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2432467627a648f1926f1ec1d5908659: Network error: Client connection negotiation failed: client connection to 127.15.216.65:39829: connect: Connection refused (error 111)
I20260810 04:16:24.680508 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20984
I20260810 04:16:24.687896 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 20601
2026-08-10T04:16:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2044 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T04:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:24Z Disabled control of system clock
I20260810 04:16:24.719727 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41357
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:16:24.806739 21174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:24.806975 21174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:24.807000 21174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:24.808158 21174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:24.808203 21174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:24.808223 21174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:24.808238 21174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:24.808249 21174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:24.809650 21174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:24.809876 21174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:24.810139 21174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:24.815454 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.815470 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.815460 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:24.815779 21174 server_base.cc:1034] running on GCE node
I20260810 04:16:24.816085 21174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:24.816430 21174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:24.817648 21174 hybrid_clock.cc:648] HybridClock initialized: now 1786335384817608 us; error 53 us; skew 500 ppm
I20260810 04:16:24.819118 21174 webserver.cc:460] Webserver started at http://127.15.216.126:33173/ using document root <none> and password file <none>
I20260810 04:16:24.819415 21174 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:24.819469 21174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:24.819638 21174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:24.820770 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "e7ea21ed03654e1289bebed614245998"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "149e924ecde2f1d69535a157137ad8e3"
server_key_iv: "f56626f5be8a382813eeb119a26341e5"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.821151 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "e7ea21ed03654e1289bebed614245998"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "149e924ecde2f1d69535a157137ad8e3"
server_key_iv: "f56626f5be8a382813eeb119a26341e5"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.822733 21174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:24.823673 21189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:24.823922 21174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:24.824045 21174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e7ea21ed03654e1289bebed614245998"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "149e924ecde2f1d69535a157137ad8e3"
server_key_iv: "f56626f5be8a382813eeb119a26341e5"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.824128 21174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:24.827819 21174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:24.828126 21174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:24.828284 21174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:24.832439 21174 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41357
I20260810 04:16:24.832522 21246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41357 every 8 connection(s)
I20260810 04:16:24.832871 21174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:24.833811 21247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:24.834853 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 21174
I20260810 04:16:24.834980 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:24.835177 16225 external_mini_cluster.cc:1428] Setting key 3eb4b864e7c8dbfcbf1f8b7d3950f2c9
I20260810 04:16:24.837841 21247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998: Bootstrap starting.
I20260810 04:16:24.838097 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:24.835826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.15.216.126:41357)
0810 04:16:24.836064 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:24.836072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:24.836097 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:16:24.836565 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:24.836572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:24.836614 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:24.836790 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:24.837557 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:24.837561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:24.837561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:24.837567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:24.837659 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:24.837673 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:24.837758 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:24.837840 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 04:16:24.837900 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20260810 04:16:24.838758 21247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:24.839182 21247 log.cc:824] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:24.839978 21247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998: No bootstrap required, opened a new log
I20260810 04:16:24.841547 21247 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ea21ed03654e1289bebed614245998" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41357 } }
I20260810 04:16:24.841826 21247 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:24.841892 21247 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ea21ed03654e1289bebed614245998, State: Initialized, Role: FOLLOWER
I20260810 04:16:24.842036 21247 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ea21ed03654e1289bebed614245998" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41357 } }
I20260810 04:16:24.842166 21247 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:24.842222 21247 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:24.842283 21247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:24.843452 21247 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ea21ed03654e1289bebed614245998" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41357 } }
I20260810 04:16:24.843604 21247 leader_election.cc:302] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ea21ed03654e1289bebed614245998; no voters:
I20260810 04:16:24.843859 21247 leader_election.cc:288] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:24.843914 21252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:24.844095 21252 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [term 1 LEADER]: Becoming Leader. State: Replica: e7ea21ed03654e1289bebed614245998, State: Running, Role: LEADER
I20260810 04:16:24.844213 21247 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:24.844198 21252 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ea21ed03654e1289bebed614245998" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41357 } }
I20260810 04:16:24.846449 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ea21ed03654e1289bebed614245998. Latest consensus state: current_term: 1 leader_uuid: "e7ea21ed03654e1289bebed614245998" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ea21ed03654e1289bebed614245998" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41357 } } }
I20260810 04:16:24.846525 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:24.846302 21253 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ea21ed03654e1289bebed614245998" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ea21ed03654e1289bebed614245998" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41357 } } }
I20260810 04:16:24.846700 21253 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998 [sys.catalog]: This master's current role is: LEADER
W20260810 04:16:24.847014 21266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:16:24.847082 21266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:16:24.847178 21268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:24.847374 21268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:24.849098 21268 catalog_manager.cc:1347] Generated new cluster ID: f0795f49f91542af892cc521c2f540c4
I20260810 04:16:24.849162 21268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:24.863278 21268 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:24.863502 21268 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:24.871471 21268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998: Generated new TSK 0
I20260810 04:16:24.871726 21268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:16:24.878063 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41357
--builtin_ntp_servers=127.15.216.84:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 04:16:24.984253 21271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:24.984467 21271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:24.984491 21271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:24.985814 21271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:24.985858 21271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:24.985885 21271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:24.987295 21271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:24.987556 21271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:24.987883 21271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:24.994444 21279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.994428 21276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:24.994428 21277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:24.994513 21271 server_base.cc:1034] running on GCE node
I20260810 04:16:24.994916 21271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:24.995180 21271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:24.996377 21271 hybrid_clock.cc:648] HybridClock initialized: now 1786335384996352 us; error 41 us; skew 500 ppm
I20260810 04:16:24.997808 21271 webserver.cc:460] Webserver started at http://127.15.216.65:37467/ using document root <none> and password file <none>
I20260810 04:16:24.998070 21271 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:24.998145 21271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:24.998358 21271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:24.999480 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "f18db6897b5feda0159ea8a4f38b88a0"
server_key_iv: "3e59425b4026e8cbbb78f9457f7a1641"
server_key_version: "encryptionkey@0"
I20260810 04:16:24.999853 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "f18db6897b5feda0159ea8a4f38b88a0"
server_key_iv: "3e59425b4026e8cbbb78f9457f7a1641"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.001367 21271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:25.002450 21285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.002651 21271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:25.002766 21271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33"
format_stamp: "Formatted at 2026-08-10 04:16:24 on dist-test-slave-g350"
server_key: "f18db6897b5feda0159ea8a4f38b88a0"
server_key_iv: "3e59425b4026e8cbbb78f9457f7a1641"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.002867 21271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:25.006438 21271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:25.006750 21271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:25.006906 21271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:25.007129 21271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:25.007467 21271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:25.007524 21271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.007572 21271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:25.007599 21271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.014437 21271 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:43171
I20260810 04:16:25.014501 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:43171 every 8 connection(s)
I20260810 04:16:25.014858 21271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:25.015429 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 21271
I20260810 04:16:25.015535 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:25.015745 16225 external_mini_cluster.cc:1428] Setting key dba79ca35175c78a3fb4828ed9a1a28a
I20260810 04:16:25.016783 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41357
--builtin_ntp_servers=127.15.216.84:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 04:16:25.018394 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.015947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:59431 (local address 127.15.216.126:41357)
0810 04:16:25.016006 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.016008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.016023 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:16:25.016642 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.016643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.016649 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.016709 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.018120 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.018123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.018123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.018128 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.018145 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.018150 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.018215 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.018276 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.018292 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:16:25.018558 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.015619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41357 (local address 127.15.216.65:59431)
0810 04:16:25.015884 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.015947 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.016525 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.016797 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.016801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.016811 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.017943 (+ 1132us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.017944 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.017989 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.018019 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.018220 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.018224 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.018227 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.018284 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.018338 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260810 04:16:25.019120 21404 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41357
I20260810 04:16:25.019212 21404 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:25.019428 21404 heartbeater.cc:511] Master 127.15.216.126:41357 requested a full tablet report, sending...
I20260810 04:16:25.020020 21208 ts_manager.cc:194] Registered new tserver with Master: 04bcb88098dd4d1ebfa1aecefbb3bd33 (127.15.216.65:43171)
W20260810 04:16:25.103327 21407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:25.103578 21407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:25.103621 21407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:25.104875 21407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:25.104950 21407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:25.104996 21407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:25.106601 21407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:25.106900 21407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:25.107220 21407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:25.112725 21414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.112723 21415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.112767 21417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:25.113035 21407 server_base.cc:1034] running on GCE node
I20260810 04:16:25.113307 21407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:25.113639 21407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:25.114799 21407 hybrid_clock.cc:648] HybridClock initialized: now 1786335385114789 us; error 30 us; skew 500 ppm
I20260810 04:16:25.116111 21407 webserver.cc:460] Webserver started at http://127.15.216.66:46159/ using document root <none> and password file <none>
I20260810 04:16:25.116425 21407 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:25.116501 21407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:25.116657 21407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:25.117847 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "25f4bfb0626248e09e31a2a10b67d379"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "ab7245d766aea7791e1f658e26a7e2ca"
server_key_iv: "73a3c73f6e56fdbafc1037d74ba5c97a"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.118239 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "25f4bfb0626248e09e31a2a10b67d379"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "ab7245d766aea7791e1f658e26a7e2ca"
server_key_iv: "73a3c73f6e56fdbafc1037d74ba5c97a"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.119628 21407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 04:16:25.120421 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.120638 21407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:25.120779 21407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "25f4bfb0626248e09e31a2a10b67d379"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "ab7245d766aea7791e1f658e26a7e2ca"
server_key_iv: "73a3c73f6e56fdbafc1037d74ba5c97a"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.120869 21407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:25.124588 21407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:25.124879 21407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:25.125022 21407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:25.125241 21407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:25.125560 21407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:25.125643 21407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.125681 21407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:25.125707 21407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.132108 21407 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:40163
I20260810 04:16:25.132206 21543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:40163 every 8 connection(s)
I20260810 04:16:25.132519 21407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:25.133081 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 21407
I20260810 04:16:25.133167 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:25.133374 16225 external_mini_cluster.cc:1428] Setting key 81586ffd4c848d5334354fa40c8dc8e0
I20260810 04:16:25.134497 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41357
--builtin_ntp_servers=127.15.216.84:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 04:16:25.135895 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.133257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:58737 (local address 127.15.216.126:41357)
0810 04:16:25.133371 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.133373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.133649 (+ 276us) server_negotiation.cc:407] Connection header received
0810 04:16:25.134134 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.134136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.134141 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.134185 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.135647 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.135649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.135649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.135653 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.135668 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.135672 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.135723 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.135825 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.135838 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 04:16:25.136164 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.133246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41357 (local address 127.15.216.66:58737)
0810 04:16:25.133503 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.133549 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.134057 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.134280 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.134285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.134295 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.135470 (+ 1175us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.135471 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.135507 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.135543 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.135738 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.135744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.135748 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.135821 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.135885 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
I20260810 04:16:25.136811 21544 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41357
I20260810 04:16:25.136889 21544 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:25.137050 21544 heartbeater.cc:511] Master 127.15.216.126:41357 requested a full tablet report, sending...
I20260810 04:16:25.137529 21208 ts_manager.cc:194] Registered new tserver with Master: 25f4bfb0626248e09e31a2a10b67d379 (127.15.216.66:40163)
W20260810 04:16:25.219233 21547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:25.219568 21547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:25.219607 21547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:25.220779 21547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:25.220845 21547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:25.220890 21547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:25.222260 21547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:25.222518 21547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:25.222803 21547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:25.228334 21555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.228343 21553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.228366 21552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:25.228607 21547 server_base.cc:1034] running on GCE node
I20260810 04:16:25.228837 21547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:25.229092 21547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:25.230255 21547 hybrid_clock.cc:648] HybridClock initialized: now 1786335385230246 us; error 31 us; skew 500 ppm
I20260810 04:16:25.231545 21547 webserver.cc:460] Webserver started at http://127.15.216.67:41679/ using document root <none> and password file <none>
I20260810 04:16:25.231807 21547 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:25.231879 21547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:25.232038 21547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:25.233170 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "bf8b2c0bb42a49cba2990a79df71f8a2"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "65eb05c57dc27918a43c8958e036bb64"
server_key_iv: "19fcfa368e16f74ddb7655c2236ac89b"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.233567 21547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "bf8b2c0bb42a49cba2990a79df71f8a2"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "65eb05c57dc27918a43c8958e036bb64"
server_key_iv: "19fcfa368e16f74ddb7655c2236ac89b"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.235031 21547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:25.235885 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.236076 21547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:25.236183 21547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "bf8b2c0bb42a49cba2990a79df71f8a2"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "65eb05c57dc27918a43c8958e036bb64"
server_key_iv: "19fcfa368e16f74ddb7655c2236ac89b"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.236259 21547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:25.239770 21547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:25.240053 21547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:25.240191 21547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:25.240427 21547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:25.240737 21547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:25.240789 21547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.240821 21547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:25.240849 21547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.247512 21547 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:37205
I20260810 04:16:25.247560 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:37205 every 8 connection(s)
I20260810 04:16:25.247892 21547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:25.250864 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 21547
I20260810 04:16:25.251008 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:25.251163 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.248730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:54531 (local address 127.15.216.126:41357)
0810 04:16:25.248793 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.248794 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.249041 (+ 247us) server_negotiation.cc:407] Connection header received
0810 04:16:25.249492 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.249494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.249499 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.249567 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.250825 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.250828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.250828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.250833 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.250851 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.250856 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.250927 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.251073 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.251092 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:16:25.251197 16225 external_mini_cluster.cc:1428] Setting key 4fc12fef57e853328e16a372ca1c914e
I20260810 04:16:25.251358 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.248681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41357 (local address 127.15.216.67:54531)
0810 04:16:25.248902 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.248945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.249404 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.249702 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.249705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.249713 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.250639 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.250640 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.250669 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.250697 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.250971 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.250978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.250982 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.251064 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.251127 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":114,"threads_started":1}
I20260810 04:16:25.251977 21686 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41357
I20260810 04:16:25.252079 21686 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:25.252233 21686 heartbeater.cc:511] Master 127.15.216.126:41357 requested a full tablet report, sending...
I20260810 04:16:25.252612 21208 ts_manager.cc:194] Registered new tserver with Master: bf8b2c0bb42a49cba2990a79df71f8a2 (127.15.216.67:37205)
I20260810 04:16:25.263283 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:16:25.266260 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.264746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.15.216.126:41357)
0810 04:16:25.264808 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.264810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.264947 (+ 137us) server_negotiation.cc:407] Connection header received
0810 04:16:25.265054 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.265055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.265060 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.265109 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.266004 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.266006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.266006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.266010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.266027 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.266032 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.266108 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.266153 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.266169 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:16:25.268045 21208 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:25.269771 21208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:25.274056 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.272272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:37205 (local address 127.0.0.1:48044)
0810 04:16:25.272447 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.272491 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.272553 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.273009 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.273010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.273012 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.273537 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.273538 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.273543 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.273554 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.273854 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.273855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.273856 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.273889 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.273903 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260810 04:16:25.274107 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.272556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.15.216.66:40163)
0810 04:16:25.272713 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.272718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.272730 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:16:25.272757 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.272759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.272770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.272856 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.273631 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.273641 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.273641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.273647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.273688 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.273721 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.273958 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.273983 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.274001 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260810 04:16:25.274056 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.272288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:40163 (local address 127.0.0.1:33070)
0810 04:16:25.272483 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.272496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.272554 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.272870 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.272877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.272883 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.273432 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.273433 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.273475 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.273503 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.273808 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.273816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.273819 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.273870 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.273885 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260810 04:16:25.274624 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.272306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:43171 (local address 127.0.0.1:40624)
0810 04:16:25.272652 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.272662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.272695 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.273750 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.273751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.273753 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.274293 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.274294 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.274300 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.274312 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.274493 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.274493 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.274494 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.274543 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.274557 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":91,"threads_started":1}
I20260810 04:16:25.274804 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.272586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.15.216.67:37205)
0810 04:16:25.272839 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.272844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.272855 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:25.272880 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.272882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.272891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.273150 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.273663 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.273670 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.273670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.273675 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.273741 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.273768 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.274679 (+ 911us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.274704 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.274721 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260810 04:16:25.275281 21474 tablet_service.cc:1467] Processing CreateTablet for tablet 590f1339dc2343e8a70a9975cdc3e5d9 (DEFAULT_TABLE table=test-table [id=d1688f13ed7f4d47a1c2a2b77d85251b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:25.275373 21616 tablet_service.cc:1467] Processing CreateTablet for tablet 590f1339dc2343e8a70a9975cdc3e5d9 (DEFAULT_TABLE table=test-table [id=d1688f13ed7f4d47a1c2a2b77d85251b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:25.275677 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590f1339dc2343e8a70a9975cdc3e5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:25.275743 21616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590f1339dc2343e8a70a9975cdc3e5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:25.277733 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.272753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.15.216.65:43171)
0810 04:16:25.273568 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.273573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.273586 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:16:25.273635 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.273638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.273649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.273736 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.274384 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.274388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.274388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.274391 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.274415 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.274427 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.274480 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.274548 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.274560 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":120,"threads_started":1}
I20260810 04:16:25.278594 21700 tablet_bootstrap.cc:492] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Bootstrap starting.
I20260810 04:16:25.278555 21334 tablet_service.cc:1467] Processing CreateTablet for tablet 590f1339dc2343e8a70a9975cdc3e5d9 (DEFAULT_TABLE table=test-table [id=d1688f13ed7f4d47a1c2a2b77d85251b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:25.278929 21334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590f1339dc2343e8a70a9975cdc3e5d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:25.279317 21700 tablet_bootstrap.cc:654] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:25.279652 21700 log.cc:824] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:25.282011 21702 tablet_bootstrap.cc:492] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33: Bootstrap starting.
I20260810 04:16:25.282105 21703 tablet_bootstrap.cc:492] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Bootstrap starting.
I20260810 04:16:25.282743 21702 tablet_bootstrap.cc:654] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:25.282840 21703 tablet_bootstrap.cc:654] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:25.282840 21700 tablet_bootstrap.cc:492] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: No bootstrap required, opened a new log
I20260810 04:16:25.282919 21700 ts_tablet_manager.cc:1397] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:16:25.283094 21702 log.cc:824] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:25.283171 21703 log.cc:824] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:25.284050 21702 tablet_bootstrap.cc:492] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33: No bootstrap required, opened a new log
I20260810 04:16:25.284164 21702 ts_tablet_manager.cc:1397] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:25.285215 21700 raft_consensus.cc:348] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.285351 21700 raft_consensus.cc:374] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:25.285373 21700 raft_consensus.cc:729] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf8b2c0bb42a49cba2990a79df71f8a2, State: Initialized, Role: FOLLOWER
I20260810 04:16:25.285501 21700 consensus_queue.cc:260] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.285825 21700 ts_tablet_manager.cc:1428] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:25.285879 21686 heartbeater.cc:503] Master 127.15.216.126:41357 was elected leader, sending a full tablet report...
I20260810 04:16:25.286274 21703 tablet_bootstrap.cc:492] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: No bootstrap required, opened a new log
I20260810 04:16:25.286356 21703 ts_tablet_manager.cc:1397] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:16:25.286605 21702 raft_consensus.cc:348] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.286759 21702 raft_consensus.cc:374] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:25.286810 21702 raft_consensus.cc:729] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bcb88098dd4d1ebfa1aecefbb3bd33, State: Initialized, Role: FOLLOWER
I20260810 04:16:25.286936 21702 consensus_queue.cc:260] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.287225 21702 ts_tablet_manager.cc:1428] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:25.288574 21703 raft_consensus.cc:348] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.288718 21703 raft_consensus.cc:374] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:25.288764 21703 raft_consensus.cc:729] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f4bfb0626248e09e31a2a10b67d379, State: Initialized, Role: FOLLOWER
I20260810 04:16:25.288908 21703 consensus_queue.cc:260] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.289171 21703 ts_tablet_manager.cc:1428] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:25.289239 21544 heartbeater.cc:503] Master 127.15.216.126:41357 was elected leader, sending a full tablet report...
I20260810 04:16:25.289295 21404 heartbeater.cc:503] Master 127.15.216.126:41357 was elected leader, sending a full tablet report...
I20260810 04:16:25.304291 21708 raft_consensus.cc:482] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:25.304423 21708 raft_consensus.cc:504] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.304782 21708 leader_election.cc:288] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04bcb88098dd4d1ebfa1aecefbb3bd33 (127.15.216.65:43171), bf8b2c0bb42a49cba2990a79df71f8a2 (127.15.216.67:37205)
I20260810 04:16:25.306316 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.304986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:43171 (local address 127.15.216.66:40123)
0810 04:16:25.305047 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.305059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.305099 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.305263 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.305264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.305266 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.305828 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.305829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.305838 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.305853 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.306105 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.306106 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.306107 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.306185 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.306200 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:16:25.306411 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.305095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:40123 (local address 127.15.216.65:43171)
0810 04:16:25.305143 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.305145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.305153 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:25.305181 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.305182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.305187 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.305249 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.305971 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.305977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.305977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.305982 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.306016 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.306022 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.306102 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.306311 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.306327 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:16:25.306448 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.305169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:48255 (local address 127.15.216.67:37205)
0810 04:16:25.305222 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.305290 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.305302 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:16:25.305339 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.305340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.305347 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.305417 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.306194 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.306195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.306196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.306201 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.306219 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.306224 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.306278 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.306364 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.306379 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:16:25.306655 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.304989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:37205 (local address 127.15.216.66:48255)
0810 04:16:25.305220 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.305231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.305279 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.305434 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.305435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.305438 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:25.306061 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:25.306061 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:25.306069 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:25.306083 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:25.306305 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:25.306306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:25.306306 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 04:16:25.306357 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:16:25.306571 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":14,"thread_start_us":116,"threads_started":1}
I20260810 04:16:25.306721 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590f1339dc2343e8a70a9975cdc3e5d9" candidate_uuid: "25f4bfb0626248e09e31a2a10b67d379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" is_pre_election: true
I20260810 04:16:25.306886 21355 raft_consensus.cc:2454] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25f4bfb0626248e09e31a2a10b67d379 in term 0.
I20260810 04:16:25.307099 21637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590f1339dc2343e8a70a9975cdc3e5d9" candidate_uuid: "25f4bfb0626248e09e31a2a10b67d379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" is_pre_election: true
I20260810 04:16:25.307135 21429 leader_election.cc:302] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04bcb88098dd4d1ebfa1aecefbb3bd33, 25f4bfb0626248e09e31a2a10b67d379; no voters:
I20260810 04:16:25.307217 21637 raft_consensus.cc:2454] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25f4bfb0626248e09e31a2a10b67d379 in term 0.
I20260810 04:16:25.307276 21708 raft_consensus.cc:2781] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:25.307339 21708 raft_consensus.cc:482] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:25.307369 21708 raft_consensus.cc:3037] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:25.308128 21708 raft_consensus.cc:504] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.308292 21708 leader_election.cc:288] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [CANDIDATE]: Term 1 election: Requested vote from peers 04bcb88098dd4d1ebfa1aecefbb3bd33 (127.15.216.65:43171), bf8b2c0bb42a49cba2990a79df71f8a2 (127.15.216.67:37205)
I20260810 04:16:25.308463 21637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590f1339dc2343e8a70a9975cdc3e5d9" candidate_uuid: "25f4bfb0626248e09e31a2a10b67d379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2"
I20260810 04:16:25.308496 21355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590f1339dc2343e8a70a9975cdc3e5d9" candidate_uuid: "25f4bfb0626248e09e31a2a10b67d379" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33"
I20260810 04:16:25.308570 21637 raft_consensus.cc:3037] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:25.308598 21355 raft_consensus.cc:3037] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:25.309289 21637 raft_consensus.cc:2454] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25f4bfb0626248e09e31a2a10b67d379 in term 1.
I20260810 04:16:25.309520 21428 leader_election.cc:302] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f4bfb0626248e09e31a2a10b67d379, bf8b2c0bb42a49cba2990a79df71f8a2; no voters:
I20260810 04:16:25.309532 21355 raft_consensus.cc:2454] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25f4bfb0626248e09e31a2a10b67d379 in term 1.
I20260810 04:16:25.309633 21708 raft_consensus.cc:2781] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:25.309808 21708 raft_consensus.cc:686] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 1 LEADER]: Becoming Leader. State: Replica: 25f4bfb0626248e09e31a2a10b67d379, State: Running, Role: LEADER
I20260810 04:16:25.309900 21708 consensus_queue.cc:237] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } }
I20260810 04:16:25.310829 21207 catalog_manager.cc:5495] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25f4bfb0626248e09e31a2a10b67d379 (127.15.216.66). New cstate: current_term: 1 leader_uuid: "25f4bfb0626248e09e31a2a10b67d379" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25f4bfb0626248e09e31a2a10b67d379" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 40163 } health_report { overall_health: HEALTHY } } }
I20260810 04:16:25.324378 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.323292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.15.216.66:40163)
0810 04:16:25.323386 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.323388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.323397 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:25.323439 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.323440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.323462 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.323528 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.324134 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.324135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.324135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.324140 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.324156 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.324159 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.324213 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.324276 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.324291 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 04:16:25.326395 21637 raft_consensus.cc:1261] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 1 FOLLOWER]: Refusing update from remote peer 25f4bfb0626248e09e31a2a10b67d379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:25.326393 21355 raft_consensus.cc:1261] T 590f1339dc2343e8a70a9975cdc3e5d9 P 04bcb88098dd4d1ebfa1aecefbb3bd33 [term 1 FOLLOWER]: Refusing update from remote peer 25f4bfb0626248e09e31a2a10b67d379: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:25.326752 21708 consensus_queue.cc:1035] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04bcb88098dd4d1ebfa1aecefbb3bd33" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:25.326916 21710 consensus_queue.cc:1035] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8b2c0bb42a49cba2990a79df71f8a2" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:25.330848 21717 mvcc.cc:205] Tried to move back new op lower bound from 7316829738293719040 to 7316829738230013952. Current Snapshot: MvccSnapshot[applied={T|T < 7316829738293719040}]
I20260810 04:16:25.331631 21719 mvcc.cc:205] Tried to move back new op lower bound from 7316829738293719040 to 7316829738230013952. Current Snapshot: MvccSnapshot[applied={T|T < 7316829738293719040 or (T in {7316829738293719040})}]
I20260810 04:16:25.334491 21208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:25.334612 21208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:25.335475 21208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7ea21ed03654e1289bebed614245998: Sending DeleteTablet for 3 replicas of tablet 590f1339dc2343e8a70a9975cdc3e5d9
I20260810 04:16:25.335897 21334 tablet_service.cc:1514] Processing DeleteTablet for tablet 590f1339dc2343e8a70a9975cdc3e5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:25 UTC) from {username='slave'} at 127.0.0.1:40624
I20260810 04:16:25.335942 21474 tablet_service.cc:1514] Processing DeleteTablet for tablet 590f1339dc2343e8a70a9975cdc3e5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:25 UTC) from {username='slave'} at 127.0.0.1:33070
I20260810 04:16:25.336041 21616 tablet_service.cc:1514] Processing DeleteTablet for tablet 590f1339dc2343e8a70a9975cdc3e5d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:25 UTC) from {username='slave'} at 127.0.0.1:48044
I20260810 04:16:25.336179 21725 tablet_replica.cc:330] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: stopping tablet replica
I20260810 04:16:25.336297 21725 raft_consensus.cc:2229] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:25.336406 21727 tablet_replica.cc:330] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: stopping tablet replica
I20260810 04:16:25.336434 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 21271
I20260810 04:16:25.336581 21727 raft_consensus.cc:2229] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:25.336727 21727 raft_consensus.cc:2258] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:25.338006 21727 ts_tablet_manager.cc:1905] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:25.336491 21725 raft_consensus.cc:2258] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:25.339396 21725 ts_tablet_manager.cc:1905] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:25.340658 21727 ts_tablet_manager.cc:1918] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:25.340758 21727 log.cc:1196] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/590f1339dc2343e8a70a9975cdc3e5d9
I20260810 04:16:25.341078 21727 ts_tablet_manager.cc:1939] T 590f1339dc2343e8a70a9975cdc3e5d9 P bf8b2c0bb42a49cba2990a79df71f8a2: Deleting consensus metadata
I20260810 04:16:25.341156 21725 ts_tablet_manager.cc:1918] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:25.341228 21725 log.cc:1196] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/590f1339dc2343e8a70a9975cdc3e5d9
I20260810 04:16:25.341549 21725 ts_tablet_manager.cc:1939] T 590f1339dc2343e8a70a9975cdc3e5d9 P 25f4bfb0626248e09e31a2a10b67d379: Deleting consensus metadata
I20260810 04:16:25.341801 21192 catalog_manager.cc:4841] TS bf8b2c0bb42a49cba2990a79df71f8a2 (127.15.216.67:37205): tablet 590f1339dc2343e8a70a9975cdc3e5d9 (table test-table [id=d1688f13ed7f4d47a1c2a2b77d85251b]) successfully deleted
I20260810 04:16:25.341979 21191 catalog_manager.cc:4841] TS 25f4bfb0626248e09e31a2a10b67d379 (127.15.216.66:40163): tablet 590f1339dc2343e8a70a9975cdc3e5d9 (table test-table [id=d1688f13ed7f4d47a1c2a2b77d85251b]) successfully deleted
W20260810 04:16:25.347879 21193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:43171 (error 108)
I20260810 04:16:25.348150 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 21407
I20260810 04:16:25.348423 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.348270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:43171 (local address 127.0.0.1:40634)
0810 04:16:25.348325 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.348380 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:43171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:16:25.349920 21193 catalog_manager.cc:4584] TS 04bcb88098dd4d1ebfa1aecefbb3bd33 (127.15.216.65:43171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 590f1339dc2343e8a70a9975cdc3e5d9: Network error: Client connection negotiation failed: client connection to 127.15.216.65:43171: connect: Connection refused (error 111)
I20260810 04:16:25.357496 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 21547
I20260810 04:16:25.364027 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 21174
2026-08-10T04:16:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (675 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 04:16:25.386551 16225 mini_oidc.cc:152] Starting JWKS server
I20260810 04:16:25.387707 16225 webserver.cc:460] Webserver started at https://127.0.0.1:34789/ using document root <none> and password file <none>
I20260810 04:16:25.387789 16225 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:16:25.387919 16225 webserver.cc:460] Webserver started at http://127.0.0.1:44849/ using document root <none> and password file <none>
2026-08-10T04:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:25Z Disabled control of system clock
I20260810 04:16:25.397040 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44909
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:37257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:44909
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34789/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/testchainca.cert with env {}
W20260810 04:16:25.481153 21734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:25.481395 21734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:25.481418 21734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:25.482583 21734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:25.482623 21734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:25.482635 21734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:25.482648 21734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:25.482661 21734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:16:25.482673 21734 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:16:25.482684 21734 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34789/jwks.json
I20260810 04:16:25.484218 21734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:44909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34789/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:25.484431 21734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:25.484603 21734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:25.489774 21740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:25.489919 21734 server_base.cc:1034] running on GCE node
W20260810 04:16:25.489774 21742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.489770 21739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:25.490285 21734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:25.490595 21734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:25.491791 21734 hybrid_clock.cc:648] HybridClock initialized: now 1786335385491787 us; error 58 us; skew 500 ppm
I20260810 04:16:25.493170 21734 webserver.cc:460] Webserver started at http://127.15.216.126:43825/ using document root <none> and password file <none>
I20260810 04:16:25.493444 21734 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:25.493522 21734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:25.493700 21734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:25.494800 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "3271904f126048e7ba6530791997807d"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "7b4975a632678390c0fb18b8af8bd8ab"
server_key_iv: "0b36af51ab5567f26449f20a6d6d0c98"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.495182 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "3271904f126048e7ba6530791997807d"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "7b4975a632678390c0fb18b8af8bd8ab"
server_key_iv: "0b36af51ab5567f26449f20a6d6d0c98"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.496639 21734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:25.497546 21748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.497805 21734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:25.497925 21734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "3271904f126048e7ba6530791997807d"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "7b4975a632678390c0fb18b8af8bd8ab"
server_key_iv: "0b36af51ab5567f26449f20a6d6d0c98"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.498023 21734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:25.513206 21734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:25.513579 21734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:25.513840 21734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:25.517978 21734 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:44909
I20260810 04:16:25.518023 21807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:44909 every 8 connection(s)
I20260810 04:16:25.518406 21734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:25.519448 21808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:25.521914 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d: Bootstrap starting.
I20260810 04:16:25.522202 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 21734
I20260810 04:16:25.522316 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:25.522526 16225 external_mini_cluster.cc:1428] Setting key 51635f8c184da9baead1329285a1f281
I20260810 04:16:25.522722 21808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:25.523082 21808 log.cc:824] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:25.523826 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d: No bootstrap required, opened a new log
I20260810 04:16:25.526140 21808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3271904f126048e7ba6530791997807d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44909 } }
I20260810 04:16:25.526335 21808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:25.526379 21808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3271904f126048e7ba6530791997807d, State: Initialized, Role: FOLLOWER
I20260810 04:16:25.526513 21808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3271904f126048e7ba6530791997807d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44909 } }
I20260810 04:16:25.526600 21808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:25.526624 21808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:25.526696 21808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:25.527419 21808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3271904f126048e7ba6530791997807d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44909 } }
I20260810 04:16:25.527549 21808 leader_election.cc:302] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3271904f126048e7ba6530791997807d; no voters:
I20260810 04:16:25.527750 21808 leader_election.cc:288] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:25.527904 21813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:25.528041 21813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [term 1 LEADER]: Becoming Leader. State: Replica: 3271904f126048e7ba6530791997807d, State: Running, Role: LEADER
I20260810 04:16:25.528056 21808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:25.528134 21813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3271904f126048e7ba6530791997807d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44909 } }
I20260810 04:16:25.528446 21813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3271904f126048e7ba6530791997807d. Latest consensus state: current_term: 1 leader_uuid: "3271904f126048e7ba6530791997807d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3271904f126048e7ba6530791997807d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44909 } } }
I20260810 04:16:25.528543 21813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:25.528463 21814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3271904f126048e7ba6530791997807d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3271904f126048e7ba6530791997807d" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44909 } } }
I20260810 04:16:25.528687 21814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:25.528765 21819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:25.529006 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.523085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.15.216.126:44909)
0810 04:16:25.523385 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.523392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.523420 (+ 28us) server_negotiation.cc:407] Connection header received
0810 04:16:25.523843 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.523850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.523953 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.524082 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.524376 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.525220 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.526459 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.527062 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.527165 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:25.527977 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.527981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.527982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.528193 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.528256 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.528274 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.528367 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.528690 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.528788 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":102,"threads_started":1}
I20260810 04:16:25.529264 21819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:25.530072 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:16:25.534921 21819 catalog_manager.cc:1347] Generated new cluster ID: 666b9fa483d64aedaae8a90878071a20
I20260810 04:16:25.534984 21819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:25.557821 21819 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:25.558590 21819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:25.561695 21819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3271904f126048e7ba6530791997807d: Generated new TSK 0
I20260810 04:16:25.561903 21819 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 04:16:25.585619 21839 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 04:16:25.585816 21839 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260810 04:16:25.586401 21838 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:25.570115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:44909 (local address 127.0.0.1:38144)
0810 04:16:25.570368 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.570378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.570464 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.570681 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.570682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.570685 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:16:25.570775 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:25.570776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.571737 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.571738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:25.572359 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:25.572361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.572774 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.572776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:25.572845 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:25.572874 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:16:25.585952 (+ 13078us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 04:16:25.586094 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.216.126:44909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":7,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.216.126:44909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 04:16:25.586838 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 21734
2026-08-10T04:16:25Z chronyd exiting
I20260810 04:16:25.625487 16225 test_util.cc:175] -----------------------------------------------
I20260810 04:16:25.625597 16225 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786335362448781-16225-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 04:16:25.626430 16225 mini_oidc.cc:152] Starting JWKS server
I20260810 04:16:25.627148 16225 webserver.cc:460] Webserver started at https://127.0.0.1:42605/ using document root <none> and password file <none>
I20260810 04:16:25.627219 16225 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:16:25.627396 16225 webserver.cc:460] Webserver started at http://127.0.0.1:45061/ using document root <none> and password file <none>
2026-08-10T04:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:25Z Disabled control of system clock
I20260810 04:16:25.636174 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45275
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45275
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42605/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/testchainca.cert with env {}
W20260810 04:16:25.720912 21845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:25.721159 21845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:25.721195 21845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:25.722411 21845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:25.722482 21845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:25.722503 21845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:25.722523 21845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:25.722554 21845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:16:25.722579 21845 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:16:25.722597 21845 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42605/jwks.json
I20260810 04:16:25.724129 21845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42605/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:25.724408 21845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:25.724625 21845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:25.729555 21851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.729581 21853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.729589 21850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:25.729812 21845 server_base.cc:1034] running on GCE node
I20260810 04:16:25.730055 21845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:25.730337 21845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:25.731501 21845 hybrid_clock.cc:648] HybridClock initialized: now 1786335385731481 us; error 41 us; skew 500 ppm
I20260810 04:16:25.732755 21845 webserver.cc:460] Webserver started at http://127.15.216.126:44127/ using document root <none> and password file <none>
I20260810 04:16:25.733036 21845 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:25.733107 21845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:25.733255 21845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:25.734357 21845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "b737644441ab4eb0a4aa10fbd80c6a45"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "eafb4178535ce1171f3149ec2c5f2cc0"
server_key_iv: "a74f98592663acdde957111109e8cab1"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.734721 21845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "b737644441ab4eb0a4aa10fbd80c6a45"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "eafb4178535ce1171f3149ec2c5f2cc0"
server_key_iv: "a74f98592663acdde957111109e8cab1"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.736196 21845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:25.737016 21859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.737196 21845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:25.737344 21845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "b737644441ab4eb0a4aa10fbd80c6a45"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "eafb4178535ce1171f3149ec2c5f2cc0"
server_key_iv: "a74f98592663acdde957111109e8cab1"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.737429 21845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:25.752223 21845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:25.752580 21845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:25.752744 21845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:25.757210 21845 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:45275
I20260810 04:16:25.757265 21915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:45275 every 8 connection(s)
I20260810 04:16:25.757714 21845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:25.758486 21916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:25.761270 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 21845
I20260810 04:16:25.761300 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45: Bootstrap starting.
I20260810 04:16:25.761400 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:25.761629 16225 external_mini_cluster.cc:1428] Setting key c0d16b527976cb3d351b63c6067506ea
I20260810 04:16:25.762082 21916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:25.762458 21916 log.cc:824] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:25.763298 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45: No bootstrap required, opened a new log
I20260810 04:16:25.765467 21916 raft_consensus.cc:348] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45275 } }
I20260810 04:16:25.765682 21916 raft_consensus.cc:374] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:25.765739 21916 raft_consensus.cc:729] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b737644441ab4eb0a4aa10fbd80c6a45, State: Initialized, Role: FOLLOWER
I20260810 04:16:25.765877 21916 consensus_queue.cc:260] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45275 } }
I20260810 04:16:25.765964 21916 raft_consensus.cc:388] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:25.766012 21916 raft_consensus.cc:482] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:25.766065 21916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:25.766857 21916 raft_consensus.cc:504] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45275 } }
I20260810 04:16:25.766980 21916 leader_election.cc:302] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b737644441ab4eb0a4aa10fbd80c6a45; no voters:
I20260810 04:16:25.767163 21916 leader_election.cc:288] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:25.767216 21921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:25.767390 21921 raft_consensus.cc:686] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [term 1 LEADER]: Becoming Leader. State: Replica: b737644441ab4eb0a4aa10fbd80c6a45, State: Running, Role: LEADER
I20260810 04:16:25.767472 21916 sys_catalog.cc:564] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:25.767496 21921 consensus_queue.cc:237] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45275 } }
I20260810 04:16:25.767843 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b737644441ab4eb0a4aa10fbd80c6a45. Latest consensus state: current_term: 1 leader_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45275 } } }
I20260810 04:16:25.767922 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:25.767876 21922 sys_catalog.cc:455] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b737644441ab4eb0a4aa10fbd80c6a45" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45275 } } }
I20260810 04:16:25.768101 21922 sys_catalog.cc:458] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:25.768148 21927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:25.768703 21927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:25.768139 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.762171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.15.216.126:45275)
0810 04:16:25.762390 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.762397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.762422 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:16:25.763053 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.763061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.763120 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.763245 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.763511 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.764406 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.765582 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.766245 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.766350 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:25.767094 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:25.767099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:25.767099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:25.767426 (+ 327us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:25.767470 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:25.767486 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:25.767783 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:25.767818 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 04:16:25.767900 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260810 04:16:25.770058 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:16:25.771174 21927 catalog_manager.cc:1347] Generated new cluster ID: 0eebb3a168144b24beb7b34e934fb57f
I20260810 04:16:25.771229 21927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:25.779431 21927 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:25.780022 21927 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:25.785921 21927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b737644441ab4eb0a4aa10fbd80c6a45: Generated new TSK 0
I20260810 04:16:25.786162 21927 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 04:16:25.817951 21948 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 04:16:25.818053 21948 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 04:16:25.818396 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.795865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.15.216.126:45275)
0810 04:16:25.795948 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.795950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.796032 (+ 82us) server_negotiation.cc:407] Connection header received
0810 04:16:25.796108 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.796109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.796120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.796186 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:16:25.796339 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.796875 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.798085 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.798294 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.798361 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:25.798562 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:16:25.818153 (+ 19591us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 04:16:25.818336 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 04:16:25.818475 21919 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38908: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 04:16:25.818559 21947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:25.795758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45275 (local address 127.0.0.1:38908)
0810 04:16:25.795944 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:25.795958 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:25.796032 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:25.796198 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:25.796199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:25.796201 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:16:25.796268 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:25.796270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.796964 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.796966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:25.797984 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:25.797989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.798383 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:25.798384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:25.798450 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:25.798457 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:16:25.818310 (+ 19853us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 04:16:25.818458 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.216.126:45275: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260810 04:16:25.818924 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 21845
2026-08-10T04:16:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 04:16:25.859444 16225 mini_oidc.cc:152] Starting JWKS server
I20260810 04:16:25.860183 16225 webserver.cc:460] Webserver started at https://127.0.0.1:34615/ using document root <none> and password file <none>
I20260810 04:16:25.860260 16225 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 04:16:25.860424 16225 webserver.cc:460] Webserver started at http://127.0.0.1:34057/ using document root <none> and password file <none>
2026-08-10T04:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:25Z Disabled control of system clock
I20260810 04:16:25.870510 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45535
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:39523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34615/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 04:16:25.956934 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:25.957156 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:25.957178 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:25.958436 21954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:25.958485 21954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:25.958498 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:25.958511 21954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:25.958523 21954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:16:25.958535 21954 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:16:25.958546 21954 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34615/jwks.json
I20260810 04:16:25.959883 21954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34615/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:25.960098 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:25.960340 21954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:25.965394 21960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.965404 21962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:25.965404 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:25.965816 21954 server_base.cc:1034] running on GCE node
I20260810 04:16:25.966121 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:25.966447 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:25.967617 21954 hybrid_clock.cc:648] HybridClock initialized: now 1786335385967604 us; error 48 us; skew 500 ppm
I20260810 04:16:25.969027 21954 webserver.cc:460] Webserver started at http://127.15.216.126:35537/ using document root <none> and password file <none>
I20260810 04:16:25.969312 21954 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:25.969399 21954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:25.969553 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:25.970697 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "99b6173e455b41a2a2775367fd9a32ff"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "c301ae3ef0be16ecb115a83668896917"
server_key_iv: "d63ad51817944bbc57f904412b73a028"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.971076 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "99b6173e455b41a2a2775367fd9a32ff"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "c301ae3ef0be16ecb115a83668896917"
server_key_iv: "d63ad51817944bbc57f904412b73a028"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.972553 21954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:16:25.973398 21968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:25.973623 21954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:25.973753 21954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "99b6173e455b41a2a2775367fd9a32ff"
format_stamp: "Formatted at 2026-08-10 04:16:25 on dist-test-slave-g350"
server_key: "c301ae3ef0be16ecb115a83668896917"
server_key_iv: "d63ad51817944bbc57f904412b73a028"
server_key_version: "encryptionkey@0"
I20260810 04:16:25.973834 21954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:25.985733 21954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:25.986114 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:25.986284 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:25.990718 21954 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:45535
I20260810 04:16:25.990757 22023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:45535 every 8 connection(s)
I20260810 04:16:25.991176 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:25.991879 22024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:25.994951 22024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff: Bootstrap starting.
I20260810 04:16:25.995491 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 21954
I20260810 04:16:25.995623 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:25.995790 22024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:25.995803 16225 external_mini_cluster.cc:1428] Setting key e92b8414da943cc69b3f821c42a3433d
I20260810 04:16:25.996167 22024 log.cc:824] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:25.997080 22024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff: No bootstrap required, opened a new log
I20260810 04:16:25.999884 22024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b6173e455b41a2a2775367fd9a32ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45535 } }
I20260810 04:16:26.000068 22024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:26.000159 22024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b6173e455b41a2a2775367fd9a32ff, State: Initialized, Role: FOLLOWER
I20260810 04:16:26.000316 22024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b6173e455b41a2a2775367fd9a32ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45535 } }
I20260810 04:16:26.000412 22024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:26.000458 22024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:26.000510 22024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:26.001149 22024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b6173e455b41a2a2775367fd9a32ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45535 } }
I20260810 04:16:26.001298 22024 leader_election.cc:302] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99b6173e455b41a2a2775367fd9a32ff; no voters:
I20260810 04:16:26.001494 22024 leader_election.cc:288] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:26.001551 22029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:26.001751 22029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [term 1 LEADER]: Becoming Leader. State: Replica: 99b6173e455b41a2a2775367fd9a32ff, State: Running, Role: LEADER
I20260810 04:16:26.001827 22024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:26.001857 22029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b6173e455b41a2a2775367fd9a32ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45535 } }
I20260810 04:16:26.002224 22029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99b6173e455b41a2a2775367fd9a32ff. Latest consensus state: current_term: 1 leader_uuid: "99b6173e455b41a2a2775367fd9a32ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b6173e455b41a2a2775367fd9a32ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45535 } } }
I20260810 04:16:26.002316 22029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:26.002245 22030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99b6173e455b41a2a2775367fd9a32ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b6173e455b41a2a2775367fd9a32ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45535 } } }
I20260810 04:16:26.002532 22030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:26.002972 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:25.996357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.15.216.126:45535)
0810 04:16:25.996555 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:25.996563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:25.996592 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:16:25.997325 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:25.997335 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:25.997420 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:25.997623 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:25.997890 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:25.999509 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.000467 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.000943 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.001082 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:26.002015 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:26.002019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:26.002019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:26.002239 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:26.002282 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:26.002356 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:26.002465 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:26.002709 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 04:16:26.002798 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260810 04:16:26.004029 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 04:16:26.004230 22044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:16:26.004305 22044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:16:26.004371 22035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:26.004578 22035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:26.006392 22035 catalog_manager.cc:1347] Generated new cluster ID: 98e85075381b4cb08d988d05d413f950
I20260810 04:16:26.006460 22035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:26.030927 22035 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:26.031641 22035 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:26.042940 22035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99b6173e455b41a2a2775367fd9a32ff: Generated new TSK 0
I20260810 04:16:26.043169 22035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:16:26.062166 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:26.058322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.15.216.126:45535)
0810 04:16:26.058390 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:26.058392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:26.058548 (+ 156us) server_negotiation.cc:407] Connection header received
0810 04:16:26.058627 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:26.058628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:26.058641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:26.058719 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:16:26.058928 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.059631 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.060512 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.060723 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.060758 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:26.060975 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:16:26.061954 (+ 979us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 04:16:26.062093 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 04:16:26.062243 22028 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 04:16:26.064116 22054 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:26.058283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45535 (local address 127.0.0.1:55682)
0810 04:16:26.058467 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:26.058478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:26.058550 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:26.058761 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:26.058763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:26.058765 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:16:26.058845 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:26.058848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.059743 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.059745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:26.060359 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:26.060361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.060824 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.060825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:26.060873 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:26.060877 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:16:26.062059 (+ 1182us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 04:16:26.064019 (+ 1960us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.216.126:45535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34615/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":1835,"thread_start_us":127,"threads_started":1}
I20260810 04:16:26.064520 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 21954
E20260810 04:16:26.076597 22055 webserver.cc:533] Webserver: sslize failed:
E20260810 04:16:26.076686 22055 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T04:16:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T04:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:26Z Disabled control of system clock
I20260810 04:16:26.115756 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:38321
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:41885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:38321
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/testchainca.cert with env {}
W20260810 04:16:26.200628 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:26.200852 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:26.200877 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:26.202035 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:26.202075 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:26.202088 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:26.202100 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:26.202113 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 04:16:26.202126 22059 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 04:16:26.202137 22059 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 04:16:26.203685 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:38321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:38321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:26.203898 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:26.204072 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:26.209213 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:26.209256 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:26.209323 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:26.209765 22059 server_base.cc:1034] running on GCE node
I20260810 04:16:26.210067 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:26.210393 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:26.211596 22059 hybrid_clock.cc:648] HybridClock initialized: now 1786335386211580 us; error 47 us; skew 500 ppm
I20260810 04:16:26.213079 22059 webserver.cc:460] Webserver started at http://127.15.216.126:39993/ using document root <none> and password file <none>
I20260810 04:16:26.213411 22059 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:26.213495 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:26.213682 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:26.214838 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "616026bfc0184c788869a8d71c3dc8e0"
format_stamp: "Formatted at 2026-08-10 04:16:26 on dist-test-slave-g350"
server_key: "069e15efa6819471bd5eca6f5d90b6de"
server_key_iv: "cf42810a19a11b26163b2f1eef5decb5"
server_key_version: "encryptionkey@0"
I20260810 04:16:26.215240 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "616026bfc0184c788869a8d71c3dc8e0"
format_stamp: "Formatted at 2026-08-10 04:16:26 on dist-test-slave-g350"
server_key: "069e15efa6819471bd5eca6f5d90b6de"
server_key_iv: "cf42810a19a11b26163b2f1eef5decb5"
server_key_version: "encryptionkey@0"
I20260810 04:16:26.216713 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:26.217525 22073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:26.217722 22059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:26.217842 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "616026bfc0184c788869a8d71c3dc8e0"
format_stamp: "Formatted at 2026-08-10 04:16:26 on dist-test-slave-g350"
server_key: "069e15efa6819471bd5eca6f5d90b6de"
server_key_iv: "cf42810a19a11b26163b2f1eef5decb5"
server_key_version: "encryptionkey@0"
I20260810 04:16:26.217922 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:26.231245 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:26.231632 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:26.231817 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:26.236528 22059 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:38321
I20260810 04:16:26.236578 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:38321 every 8 connection(s)
I20260810 04:16:26.237000 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:26.238184 22130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:26.240581 22130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0: Bootstrap starting.
I20260810 04:16:26.240824 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 22059
I20260810 04:16:26.240938 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:26.241158 16225 external_mini_cluster.cc:1428] Setting key 2cb43fc58cabbe5b9774e04577ba9cf4
I20260810 04:16:26.241382 22130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:26.241806 22130 log.cc:824] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:26.242527 22130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0: No bootstrap required, opened a new log
I20260810 04:16:26.244411 22130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616026bfc0184c788869a8d71c3dc8e0" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38321 } }
I20260810 04:16:26.244571 22130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:26.244620 22130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616026bfc0184c788869a8d71c3dc8e0, State: Initialized, Role: FOLLOWER
I20260810 04:16:26.244752 22130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616026bfc0184c788869a8d71c3dc8e0" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38321 } }
I20260810 04:16:26.244872 22130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:26.244920 22130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:26.244968 22130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:26.245509 22130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616026bfc0184c788869a8d71c3dc8e0" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38321 } }
I20260810 04:16:26.245658 22130 leader_election.cc:302] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 616026bfc0184c788869a8d71c3dc8e0; no voters:
I20260810 04:16:26.245837 22130 leader_election.cc:288] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:26.245903 22135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:26.246111 22130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:26.246291 22135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [term 1 LEADER]: Becoming Leader. State: Replica: 616026bfc0184c788869a8d71c3dc8e0, State: Running, Role: LEADER
I20260810 04:16:26.246400 22135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616026bfc0184c788869a8d71c3dc8e0" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38321 } }
I20260810 04:16:26.247139 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:26.241760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.15.216.126:38321)
0810 04:16:26.241984 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:26.241994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:26.242032 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:16:26.242507 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:26.242514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:26.242574 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:26.242749 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:26.243000 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.243931 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.244796 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.245334 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.245488 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:26.246322 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:26.246326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:26.246327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:26.246532 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:26.246579 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:26.246595 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:26.246674 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:26.246779 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:16:26.246885 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260810 04:16:26.247306 22136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "616026bfc0184c788869a8d71c3dc8e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616026bfc0184c788869a8d71c3dc8e0" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38321 } } }
I20260810 04:16:26.247418 22136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:26.247671 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 616026bfc0184c788869a8d71c3dc8e0. Latest consensus state: current_term: 1 leader_uuid: "616026bfc0184c788869a8d71c3dc8e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "616026bfc0184c788869a8d71c3dc8e0" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 38321 } } }
I20260810 04:16:26.247681 22150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:26.248044 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:26.248194 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:16:26.248636 22150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:26.250386 22150 catalog_manager.cc:1347] Generated new cluster ID: 29d29c656e104c8b916625de686eee5d
I20260810 04:16:26.250443 22150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:26.257289 22150 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:26.257934 22150 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:26.267792 22150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 616026bfc0184c788869a8d71c3dc8e0: Generated new TSK 0
I20260810 04:16:26.268041 22150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:16:26.296577 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:26.274989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.15.216.126:38321)
0810 04:16:26.275043 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:26.275045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:26.275108 (+ 63us) server_negotiation.cc:407] Connection header received
0810 04:16:26.275209 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:26.275210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:26.275220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:26.275323 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0810 04:16:26.275451 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.275990 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.277157 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.277365 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.277427 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:26.277678 (+ 251us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 04:16:26.296343 (+ 18665us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 04:16:26.296514 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38216: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:16:26.296675 22133 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38216: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 04:16:26.296778 22159 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:26.274844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:38321 (local address 127.0.0.1:38216)
0810 04:16:26.275040 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:26.275053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:26.275132 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:26.275305 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:26.275306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:26.275308 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 04:16:26.275377 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:26.275379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.276077 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.276078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:26.277053 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:26.277056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:26.277453 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:26.277454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:26.277520 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:26.277525 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 04:16:26.296507 (+ 18982us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 04:16:26.296661 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.216.126:38321: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260810 04:16:26.297017 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 22059
2026-08-10T04:16:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (211 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (96 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:26 dist-test-slave-g350 krb5kdc[22174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:26 dist-test-slave-g350 krb5kdc[22174](info): set up 2 sockets
Aug 10 04:16:26 dist-test-slave-g350 krb5kdc[22174](info): commencing operation
krb5kdc: starting...
W20260810 04:16:27.583678 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:27 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:27.713842 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36585
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:36585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:27.796101 22187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:27.796310 22187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:27.796334 22187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:27.797421 22187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:27.797453 22187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:27.797467 22187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:27.797479 22187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:27.797492 22187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:27.798887 22187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:36585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:27.799081 22187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:27.799249 22187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:27.804291 22195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:27.804303 22193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:27.804303 22192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:27.804335 22187 server_base.cc:1034] running on GCE node
I20260810 04:16:27.804863 22187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:27.805152 22187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:27.806340 22187 hybrid_clock.cc:648] HybridClock initialized: now 1786335387806324 us; error 48 us; skew 500 ppm
Aug 10 04:16:27 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335387, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:27.812420 22187 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:27.812963 22187 webserver.cc:460] Webserver started at http://127.15.216.126:46775/ using document root <none> and password file <none>
I20260810 04:16:27.813201 22187 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:27.813278 22187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:27.813438 22187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:27.814626 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "ca4ab108661c44ac99509aadd5a8ce60"
format_stamp: "Formatted at 2026-08-10 04:16:27 on dist-test-slave-g350"
server_key: "32a65714ad635bc4afc51e19deb3cb56"
server_key_iv: "37ef55e0365586d01dd43a6b0e09b633"
server_key_version: "encryptionkey@0"
I20260810 04:16:27.815008 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "ca4ab108661c44ac99509aadd5a8ce60"
format_stamp: "Formatted at 2026-08-10 04:16:27 on dist-test-slave-g350"
server_key: "32a65714ad635bc4afc51e19deb3cb56"
server_key_iv: "37ef55e0365586d01dd43a6b0e09b633"
server_key_version: "encryptionkey@0"
I20260810 04:16:27.816411 22187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:27.817171 22202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:27.817356 22187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:27.817458 22187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "ca4ab108661c44ac99509aadd5a8ce60"
format_stamp: "Formatted at 2026-08-10 04:16:27 on dist-test-slave-g350"
server_key: "32a65714ad635bc4afc51e19deb3cb56"
server_key_iv: "37ef55e0365586d01dd43a6b0e09b633"
server_key_version: "encryptionkey@0"
I20260810 04:16:27.817545 22187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:27.845672 22187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:27.848486 22187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:27.848664 22187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:27.852970 22187 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:36585
I20260810 04:16:27.853034 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:36585 every 8 connection(s)
I20260810 04:16:27.853364 22187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:27.854249 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:27.856580 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60: Bootstrap starting.
I20260810 04:16:27.857177 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:27.857472 22257 log.cc:824] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:27.858158 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60: No bootstrap required, opened a new log
I20260810 04:16:27.859678 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 22187
I20260810 04:16:27.859788 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:27.859980 16225 external_mini_cluster.cc:1428] Setting key 188c7d3e874971ee85ef3433f499e17c
I20260810 04:16:27.860033 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4ab108661c44ac99509aadd5a8ce60" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36585 } }
I20260810 04:16:27.860232 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:27.860281 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4ab108661c44ac99509aadd5a8ce60, State: Initialized, Role: FOLLOWER
I20260810 04:16:27.860431 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4ab108661c44ac99509aadd5a8ce60" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36585 } }
I20260810 04:16:27.860532 22257 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:27.860579 22257 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:27.860631 22257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:27.861836 22257 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4ab108661c44ac99509aadd5a8ce60" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36585 } }
I20260810 04:16:27.861967 22257 leader_election.cc:302] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca4ab108661c44ac99509aadd5a8ce60; no voters:
I20260810 04:16:27.862159 22257 leader_election.cc:288] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:27.862205 22262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:27.862316 22262 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [term 1 LEADER]: Becoming Leader. State: Replica: ca4ab108661c44ac99509aadd5a8ce60, State: Running, Role: LEADER
I20260810 04:16:27.862411 22262 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4ab108661c44ac99509aadd5a8ce60" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36585 } }
I20260810 04:16:27.862454 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:27.862771 22262 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4ab108661c44ac99509aadd5a8ce60. Latest consensus state: current_term: 1 leader_uuid: "ca4ab108661c44ac99509aadd5a8ce60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4ab108661c44ac99509aadd5a8ce60" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36585 } } }
I20260810 04:16:27.862854 22262 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:27.862782 22263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca4ab108661c44ac99509aadd5a8ce60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4ab108661c44ac99509aadd5a8ce60" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36585 } } }
I20260810 04:16:27.863147 22263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:27.864130 22268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:27.864645 22268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 04:16:27 dist-test-slave-g350 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:27.871718 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:27.860509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.15.216.126:36585)
0810 04:16:27.860745 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:27.860760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:27.860787 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:16:27.860886 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:27.860893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:27.860957 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:27.861064 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:27.862035 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:27.863680 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:27.864837 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:27.865454 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:27.865564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:27.868286 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:27.868290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:27.868291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:27.868303 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:27.870150 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:27.870541 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:27.870544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:27.870545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:27.870653 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:27.870877 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:27.870878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:27.870878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:27.871115 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:27.871298 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:27.871399 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 04:16:27.871546 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":106,"threads_started":1}
I20260810 04:16:27.872113 22268 catalog_manager.cc:1347] Generated new cluster ID: 95e94fc067e24ada929f48a0f91b7b27
I20260810 04:16:27.872169 22268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:27.882668 22268 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:27.883249 22268 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:27.889557 22268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca4ab108661c44ac99509aadd5a8ce60: Generated new TSK 0
I20260810 04:16:27.889855 22268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:27.929253 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36585
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:28.022864 22285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:28.023097 22285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:28.023136 22285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:28.024250 22285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:28.024320 22285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:28.024366 22285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:28.025914 22285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:28.026183 22285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:28.026435 22285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:28.031919 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:28.031931 22294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:28.031962 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:28.032320 22285 server_base.cc:1034] running on GCE node
I20260810 04:16:28.032598 22285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:28.032869 22285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:28.034044 22285 hybrid_clock.cc:648] HybridClock initialized: now 1786335388034028 us; error 62 us; skew 500 ppm
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.037070 22285 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:28.037650 22285 webserver.cc:460] Webserver started at http://127.15.216.65:37751/ using document root <none> and password file <none>
I20260810 04:16:28.037909 22285 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:28.037977 22285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:28.038136 22285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:28.039218 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "bd67ca4a462d4f6da383cddb4884f092"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "e3dddf8b96230e5b6b840235b7ce9b5e"
server_key_iv: "d25899ec5555a09ed143720658111162"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.039573 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "bd67ca4a462d4f6da383cddb4884f092"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "e3dddf8b96230e5b6b840235b7ce9b5e"
server_key_iv: "d25899ec5555a09ed143720658111162"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.040918 22285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:28.041754 22301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.041949 22285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:28.042052 22285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "bd67ca4a462d4f6da383cddb4884f092"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "e3dddf8b96230e5b6b840235b7ce9b5e"
server_key_iv: "d25899ec5555a09ed143720658111162"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.042140 22285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:28.077548 22285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:28.080036 22285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:28.080222 22285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:28.080538 22285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:28.080883 22285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:28.080950 22285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.080991 22285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:28.081033 22285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.087569 22285 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:44287
I20260810 04:16:28.087731 22426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:44287 every 8 connection(s)
I20260810 04:16:28.087985 22285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:28.096163 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 22285
I20260810 04:16:28.096263 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:28.096462 16225 external_mini_cluster.cc:1428] Setting key c9f7f5a1bc09247141ae281f9de4b174
I20260810 04:16:28.098047 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.088732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:47367 (local address 127.15.216.126:36585)
0810 04:16:28.088804 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.088806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.089017 (+ 211us) server_negotiation.cc:407] Connection header received
0810 04:16:28.089977 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.089978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.089993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.090060 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.090656 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.091158 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.092220 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.092468 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.092538 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.094787 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:28.094789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:28.094789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:28.094798 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:28.096470 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.096961 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.096962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.096963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.097019 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.097284 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.097285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.097286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.097423 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:28.097508 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:28.097850 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 04:16:28.097957 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:16:28.098279 22430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.088714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36585 (local address 127.15.216.65:47367)
0810 04:16:28.088873 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:28.088916 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:28.089863 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:28.090113 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:28.090116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:28.090374 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:28.090564 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:28.090568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.091244 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.091245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:28.092079 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:28.092081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.092570 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.092571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:28.092704 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.093226 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:28.093234 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:28.094684 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:28.096572 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:28.096580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:28.096583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:28.096886 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:28.097088 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:28.097088 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:28.097089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:28.097207 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:28.097498 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:28.097506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:28.097712 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 04:16:28.097822 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 04:16:28.097983 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260810 04:16:28.098757 22427 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36585
I20260810 04:16:28.098873 22427 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:28.099076 22427 heartbeater.cc:511] Master 127.15.216.126:36585 requested a full tablet report, sending...
I20260810 04:16:28.099665 22220 ts_manager.cc:194] Registered new tserver with Master: bd67ca4a462d4f6da383cddb4884f092 (127.15.216.65:44287)
I20260810 04:16:28.100306 22220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:47367
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:28.125486 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36585
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:28.208746 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:28.209009 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:28.209036 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:28.210211 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:28.210258 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:28.210285 22433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:28.211755 22433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:28.212003 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:28.212235 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:28.217450 22442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:28.217479 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:28.217533 22433 server_base.cc:1034] running on GCE node
W20260810 04:16:28.217474 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:28.218058 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:28.218300 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:28.219468 22433 hybrid_clock.cc:648] HybridClock initialized: now 1786335388219440 us; error 48 us; skew 500 ppm
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.222312 22433 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:28.222771 22433 webserver.cc:460] Webserver started at http://127.15.216.66:43343/ using document root <none> and password file <none>
I20260810 04:16:28.223019 22433 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:28.223088 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:28.223239 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:28.224280 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "66513b9510eb473795257825ad1bde55"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "4b80b4f57229033438bc347a1eb25b69"
server_key_iv: "7401f421f49b1e0d528058cf957c6c43"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.224649 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "66513b9510eb473795257825ad1bde55"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "4b80b4f57229033438bc347a1eb25b69"
server_key_iv: "7401f421f49b1e0d528058cf957c6c43"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.226071 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:28.226805 22449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.227008 22433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:28.227118 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "66513b9510eb473795257825ad1bde55"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "4b80b4f57229033438bc347a1eb25b69"
server_key_iv: "7401f421f49b1e0d528058cf957c6c43"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.227191 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:28.236217 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:28.238113 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:28.238281 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:28.238513 22433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:28.238832 22433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:28.238888 22433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.238931 22433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:28.238960 22433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.245131 22433 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:44159
I20260810 04:16:28.245182 22570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:44159 every 8 connection(s)
I20260810 04:16:28.245489 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:28.251034 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 22433
I20260810 04:16:28.251130 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:28.251350 16225 external_mini_cluster.cc:1428] Setting key 61aa9edf5803291e12961e5034987143
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:28.255857 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.246352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:58125 (local address 127.15.216.126:36585)
0810 04:16:28.246419 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.246421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.246626 (+ 205us) server_negotiation.cc:407] Connection header received
0810 04:16:28.247594 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.247595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.247607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.247673 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.248321 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.248836 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.249966 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.250165 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.250222 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.252476 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:28.252478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:28.252479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:28.252487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:28.254205 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.254759 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.254759 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.254760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.254815 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.255127 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.255128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.255129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.255259 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:28.255349 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:28.255693 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 04:16:28.255773 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:16:28.256119 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.246321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36585 (local address 127.15.216.66:58125)
0810 04:16:28.246502 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:28.246544 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:28.247467 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:28.247752 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:28.247756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:28.248007 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:28.248193 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:28.248197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.248964 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.248968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:28.249892 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:28.249895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.250281 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.250282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:28.250383 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.250829 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:28.250837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:28.252380 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:28.254313 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:28.254322 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:28.254325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:28.254677 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:28.254885 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:28.254886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:28.254886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:28.255034 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:28.255334 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:28.255342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:28.255515 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 04:16:28.255634 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 04:16:28.255824 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260810 04:16:28.256573 22571 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36585
I20260810 04:16:28.256688 22571 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:28.256893 22571 heartbeater.cc:511] Master 127.15.216.126:36585 requested a full tablet report, sending...
I20260810 04:16:28.257335 22220 ts_manager.cc:194] Registered new tserver with Master: 66513b9510eb473795257825ad1bde55 (127.15.216.66:44159)
I20260810 04:16:28.257781 22220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:58125
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:28.281940 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36585
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:28.368427 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:28.368654 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:28.368695 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:28.369897 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:28.369956 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:28.369999 22576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:28.371469 22576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:28.371729 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:28.371968 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:28.377142 22585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:28.377161 22581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:28.377152 22582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:28.377257 22576 server_base.cc:1034] running on GCE node
I20260810 04:16:28.377529 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:28.377812 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:28.378965 22576 hybrid_clock.cc:648] HybridClock initialized: now 1786335388378951 us; error 26 us; skew 500 ppm
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.381518 22576 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:28.381978 22576 webserver.cc:460] Webserver started at http://127.15.216.67:34895/ using document root <none> and password file <none>
I20260810 04:16:28.382200 22576 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:28.382244 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:28.382408 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:28.383478 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "7f3964c8a5d14f74a33b19bf46f0a26f"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "8015759cf60560b9ef9e97319f7253f9"
server_key_iv: "5aba7431691b3f53602569f016333729"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.383849 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "7f3964c8a5d14f74a33b19bf46f0a26f"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "8015759cf60560b9ef9e97319f7253f9"
server_key_iv: "5aba7431691b3f53602569f016333729"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.385206 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:28.386059 22592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.386271 22576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:16:28.386379 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "7f3964c8a5d14f74a33b19bf46f0a26f"
format_stamp: "Formatted at 2026-08-10 04:16:28 on dist-test-slave-g350"
server_key: "8015759cf60560b9ef9e97319f7253f9"
server_key_iv: "5aba7431691b3f53602569f016333729"
server_key_version: "encryptionkey@0"
I20260810 04:16:28.386458 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:28.406749 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:28.409278 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:28.409444 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:28.409741 22576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:28.410080 22576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:28.410113 22576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.410136 22576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:28.410181 22576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:28.416265 22576 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:35349
I20260810 04:16:28.416419 22714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:35349 every 8 connection(s)
I20260810 04:16:28.416666 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:28.417402 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 22576
I20260810 04:16:28.417484 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:28.417717 16225 external_mini_cluster.cc:1428] Setting key aa3f5fb6dc2f4a93c5b4bd1bb55879d3
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:28.426441 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.417666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:59829 (local address 127.15.216.126:36585)
0810 04:16:28.417719 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.417722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.417755 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:16:28.418940 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.418941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.418954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.419009 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.419685 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.420283 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.421277 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.421549 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.421642 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.423697 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:28.423698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:28.423699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:28.423707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:28.425007 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.425431 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.425432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.425432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.425498 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.425777 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.425778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.425778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.425906 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:28.425982 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:28.426260 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 04:16:28.426343 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:16:28.426662 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.417438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36585 (local address 127.15.216.67:59829)
0810 04:16:28.417627 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:28.417676 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:28.418836 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:28.419081 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:28.419085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:28.419386 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:28.419577 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:28.419581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.420365 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.420367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:28.421150 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:28.421152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.421667 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.421668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:28.421787 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.422207 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:28.422215 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:28.423599 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:28.425096 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:28.425103 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:28.425106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:28.425350 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:28.425567 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:28.425568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:28.425569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:28.425705 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:28.425984 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:28.425992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:28.426125 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 04:16:28.426235 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 04:16:28.426409 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260810 04:16:28.427222 22715 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36585
I20260810 04:16:28.427362 22715 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:28.427515 22715 heartbeater.cc:511] Master 127.15.216.126:36585 requested a full tablet report, sending...
I20260810 04:16:28.427945 22220 ts_manager.cc:194] Registered new tserver with Master: 7f3964c8a5d14f74a33b19bf46f0a26f (127.15.216.67:35349)
I20260810 04:16:28.428308 22220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:59829
I20260810 04:16:28.429436 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/test-admin.keytab.
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.447362 16225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 04:16:28.449132 22722 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:28.448249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.15.216.65:44287)
0810 04:16:28.448451 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.448456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.448467 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:28.448493 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.448495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.448510 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.448578 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.449044 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45840: BlockingRecv error: recv got EOF from 127.0.0.1:45840 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
W20260810 04:16:28.449198 16225 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.450256 16225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.451138 16225 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:28.459648 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.451662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.15.216.65:44287)
0810 04:16:28.451715 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.451717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.451769 (+ 52us) server_negotiation.cc:407] Connection header received
0810 04:16:28.451819 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.451820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.451830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.451878 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.452234 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.452961 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.453854 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.454160 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.454216 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.456611 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:28.456620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:28.456620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:28.456627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:28.458457 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.458798 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.458801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.458801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.458861 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.459056 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.459058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.459058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.459205 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:28.459322 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:28.459388 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 04:16:28.459454 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 04:16:28.461438 22722 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:28.460959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.15.216.65:44287)
0810 04:16:28.461013 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.461014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.461045 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:16:28.461103 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.461104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.461113 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.461149 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.461406 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45846: BlockingRecv error: recv got EOF from 127.0.0.1:45846 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 04:16:28.461575 16225 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc)
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.462616 16225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.463608 16225 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:28.470597 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.464125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.15.216.65:44287)
0810 04:16:28.464162 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.464163 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.464252 (+ 89us) server_negotiation.cc:407] Connection header received
0810 04:16:28.464306 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.464307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.464319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.464388 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.464703 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.465165 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.465857 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.466068 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.466134 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.468141 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:28.468142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:28.468143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:28.468149 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:28.469697 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.469994 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.469995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.469995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.470045 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.470205 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.470206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.470206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.470322 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:28.470394 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:28.470482 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:16:28.470540 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:16:28.471999 22722 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:28.471553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.15.216.65:44287)
0810 04:16:28.471611 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.471613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.471620 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:28.471667 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.471668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.471675 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.471721 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.471948 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45860: BlockingRecv error: recv got EOF from 127.0.0.1:45860 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 04:16:28.472110 16225 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786335362448781-16225-0/krb5kdc/krb5cc' of type FILE
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.473076 16225 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:28.474018 16225 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:16:28.481390 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:28.474582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.15.216.65:44287)
0810 04:16:28.474623 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:28.474625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:28.474699 (+ 74us) server_negotiation.cc:407] Connection header received
0810 04:16:28.474749 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:28.474749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:28.474757 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:28.474804 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:28.475144 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.475605 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.476287 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:28.476490 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:28.476543 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:28.478674 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:28.478675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:28.478675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:28.478682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:28.480208 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.480622 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.480623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.480624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.480688 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:28.480931 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:28.480932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:28.480932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:28.481039 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:28.481144 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:28.481222 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 04:16:28.481323 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:16:28.482173 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 22285
I20260810 04:16:28.489697 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 22433
I20260810 04:16:28.496573 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 22576
I20260810 04:16:28.502461 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 22187
2026-08-10T04:16:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2016 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22737](info): set up 2 sockets
Aug 10 04:16:28 dist-test-slave-g350 krb5kdc[22737](info): commencing operation
krb5kdc: starting...
W20260810 04:16:29.611214 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:29 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab.
Entry for principal oryx/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab.
I20260810 04:16:29.740299 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:34993
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab
--principal=oryx/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:34993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:29.824019 22750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:29.824285 22750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:29.824323 22750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:29.825640 22750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:29.825721 22750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:29.825749 22750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:29.825780 22750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:29.825817 22750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:29.827384 22750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:34993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab
--principal=oryx/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:34993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:29.827682 22750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:29.827929 22750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:29.832986 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:29.833006 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:29.833078 22750 server_base.cc:1034] running on GCE node
W20260810 04:16:29.832983 22759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:29.833441 22750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:29.833827 22750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:29.834988 22750 hybrid_clock.cc:648] HybridClock initialized: now 1786335389834977 us; error 46 us; skew 500 ppm
Aug 10 04:16:29 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335389, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:29.837667 22750 init.cc:377] Logged in from keytab as oryx/127.15.216.126@KRBTEST.COM (short username oryx)
I20260810 04:16:29.838125 22750 webserver.cc:460] Webserver started at http://127.15.216.126:44729/ using document root <none> and password file <none>
I20260810 04:16:29.838363 22750 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:29.838410 22750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:29.838581 22750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:29.839656 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "3fab64e82a934b07aed27d6704f08048"
format_stamp: "Formatted at 2026-08-10 04:16:29 on dist-test-slave-g350"
server_key: "7af2d24551e4f6f69361eef7b092180b"
server_key_iv: "6e5d1a2eda5bda377ca3e43558691201"
server_key_version: "encryptionkey@0"
I20260810 04:16:29.840030 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "3fab64e82a934b07aed27d6704f08048"
format_stamp: "Formatted at 2026-08-10 04:16:29 on dist-test-slave-g350"
server_key: "7af2d24551e4f6f69361eef7b092180b"
server_key_iv: "6e5d1a2eda5bda377ca3e43558691201"
server_key_version: "encryptionkey@0"
I20260810 04:16:29.841492 22750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:29.842633 22766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:29.842895 22750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:29.843003 22750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "3fab64e82a934b07aed27d6704f08048"
format_stamp: "Formatted at 2026-08-10 04:16:29 on dist-test-slave-g350"
server_key: "7af2d24551e4f6f69361eef7b092180b"
server_key_iv: "6e5d1a2eda5bda377ca3e43558691201"
server_key_version: "encryptionkey@0"
I20260810 04:16:29.843078 22750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:29.860816 22750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:29.863009 22750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:29.863158 22750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:29.867453 22750 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:34993
I20260810 04:16:29.867492 22822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:34993 every 8 connection(s)
I20260810 04:16:29.867892 22750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:29.868963 22823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:29.871225 22823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: Bootstrap starting.
I20260810 04:16:29.871768 22823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:29.872004 22823 log.cc:824] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:29.872704 22823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: No bootstrap required, opened a new log
I20260810 04:16:29.874426 22823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } }
I20260810 04:16:29.874567 22823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:29.874593 22823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fab64e82a934b07aed27d6704f08048, State: Initialized, Role: FOLLOWER
I20260810 04:16:29.874734 22823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } }
I20260810 04:16:29.874821 22823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:29.874845 22823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:29.874877 22823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:29.875710 22823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } }
I20260810 04:16:29.875731 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 22750
I20260810 04:16:29.875801 22823 leader_election.cc:302] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fab64e82a934b07aed27d6704f08048; no voters:
I20260810 04:16:29.875850 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:29.876001 22823 leader_election.cc:288] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:29.876048 22826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:29.876112 16225 external_mini_cluster.cc:1428] Setting key 50d8f86f7bcedcdcb94bc4dd9ab83221
I20260810 04:16:29.876200 22826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 1 LEADER]: Becoming Leader. State: Replica: 3fab64e82a934b07aed27d6704f08048, State: Running, Role: LEADER
I20260810 04:16:29.876302 22823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:29.876303 22826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } }
I20260810 04:16:29.876679 22827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } }
I20260810 04:16:29.876765 22827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:29.877243 22834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:29.877345 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fab64e82a934b07aed27d6704f08048. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } }
I20260810 04:16:29.877487 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:29.877818 22834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:29.879609 22834 catalog_manager.cc:1347] Generated new cluster ID: c3fc8f39de79412296c255f63da00da1
I20260810 04:16:29.879665 22834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:16:29 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:29.887635 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:29.876812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.15.216.126:34993)
0810 04:16:29.876995 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:29.877005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:29.877166 (+ 161us) server_negotiation.cc:407] Connection header received
0810 04:16:29.877616 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:29.877626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:29.877689 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:29.877841 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:29.878646 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:29.879963 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:29.880726 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:29.881140 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:29.881261 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:29.883759 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:29.883763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:29.883763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:29.883775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:29.885892 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:29.886345 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:29.886348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:29.886349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:29.886489 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:29.886737 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:29.886738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:29.886738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:29.887001 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:29.887192 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:29.887258 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 04:16:29.887400 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260810 04:16:29.892668 22834 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:29.893186 22834 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:29.898447 22834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: Generated new TSK 0
I20260810 04:16:29.898633 22834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab.
Entry for principal oryx/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab.
I20260810 04:16:29.928082 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:34993
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab
--principal=oryx/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:30.015589 22852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:30.015820 22852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:30.015846 22852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:30.016999 22852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:30.017040 22852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:30.017068 22852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:30.018545 22852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab
--principal=oryx/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:34993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:30.018749 22852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:30.018939 22852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:30.024245 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.024263 22862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.024340 22860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:30.024514 22852 server_base.cc:1034] running on GCE node
I20260810 04:16:30.024719 22852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:30.024935 22852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:30.026098 22852 hybrid_clock.cc:648] HybridClock initialized: now 1786335390026084 us; error 28 us; skew 500 ppm
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:30.028653 22852 init.cc:377] Logged in from keytab as oryx/127.15.216.65@KRBTEST.COM (short username oryx)
I20260810 04:16:30.029100 22852 webserver.cc:460] Webserver started at http://127.15.216.65:37457/ using document root <none> and password file <none>
I20260810 04:16:30.029343 22852 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.029397 22852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.029525 22852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:30.030608 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "8ca26d5b94ef47049610a78d254d2415"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "5b2fbc480046f2306b9b7febd415104a"
server_key_iv: "7f59fb896ea2661fa4a11994e10ac725"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.030997 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "8ca26d5b94ef47049610a78d254d2415"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "5b2fbc480046f2306b9b7febd415104a"
server_key_iv: "7f59fb896ea2661fa4a11994e10ac725"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.032486 22852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:30.033339 22869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.033566 22852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:30.033699 22852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "8ca26d5b94ef47049610a78d254d2415"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "5b2fbc480046f2306b9b7febd415104a"
server_key_iv: "7f59fb896ea2661fa4a11994e10ac725"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.033785 22852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.052430 22852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:30.054710 22852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:30.054888 22852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:30.055153 22852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:30.055490 22852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:30.055544 22852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.055581 22852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:30.055629 22852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.061704 22852 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:32781
I20260810 04:16:30.061803 22993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:32781 every 8 connection(s)
I20260810 04:16:30.062124 22852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:30.063700 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 22852
I20260810 04:16:30.063829 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:30.064085 16225 external_mini_cluster.cc:1428] Setting key 710596622a6cd81a41b155c1fe3f3a60
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:30.072549 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.062931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:46071 (local address 127.15.216.126:34993)
0810 04:16:30.063026 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.063029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.063197 (+ 168us) server_negotiation.cc:407] Connection header received
0810 04:16:30.063771 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.063772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.063789 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.063843 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.065033 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.065541 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.066580 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.066787 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.066872 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.069324 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.069326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.069326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.069334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.071093 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.071513 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.071515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.071515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.071585 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.071831 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.071831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.071832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.071980 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.072059 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.072314 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.072436 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 04:16:30.072588 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.062856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:46071)
0810 04:16:30.063068 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.063117 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.063661 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.063918 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.063922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.064629 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.064936 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.064941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.065675 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.065676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.066475 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.066477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.066890 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.066891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.067024 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.067562 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.067570 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.069223 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.071186 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.071193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.071195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.071433 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.071661 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.071662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.071662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.071763 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.072052 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.072058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.072175 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.072261 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.072412 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
I20260810 04:16:30.073184 22995 heartbeater.cc:347] Connected to a master server at 127.15.216.126:34993
I20260810 04:16:30.073282 22995 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:30.073483 22995 heartbeater.cc:511] Master 127.15.216.126:34993 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.15.216.66@KRBTEST.COM; defaulting to no policy
I20260810 04:16:30.074061 22783 ts_manager.cc:194] Registered new tserver with Master: 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65:32781)
I20260810 04:16:30.074678 22783 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:46071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab.
Entry for principal oryx/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab.
I20260810 04:16:30.094177 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:34993
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab
--principal=oryx/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:30.179605 23002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:30.179806 23002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:30.179831 23002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:30.180904 23002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:30.180939 23002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:30.180965 23002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:30.182379 23002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab
--principal=oryx/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:34993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:30.182567 23002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:30.182744 23002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:30.187858 23007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.187884 23010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.187908 23008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:30.188383 23002 server_base.cc:1034] running on GCE node
I20260810 04:16:30.188593 23002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:30.188786 23002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:30.189971 23002 hybrid_clock.cc:648] HybridClock initialized: now 1786335390189947 us; error 35 us; skew 500 ppm
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:30.192621 23002 init.cc:377] Logged in from keytab as oryx/127.15.216.66@KRBTEST.COM (short username oryx)
I20260810 04:16:30.193131 23002 webserver.cc:460] Webserver started at http://127.15.216.66:45493/ using document root <none> and password file <none>
I20260810 04:16:30.193368 23002 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.193440 23002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.193640 23002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:30.194712 23002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "6888322a49638dc4b51c01be5793abe6"
server_key_iv: "96a8828bcdc9575a04f7716f4254f5ba"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.195103 23002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "6888322a49638dc4b51c01be5793abe6"
server_key_iv: "96a8828bcdc9575a04f7716f4254f5ba"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.196563 23002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:30.197400 23019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.197647 23002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:30.197764 23002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "6888322a49638dc4b51c01be5793abe6"
server_key_iv: "96a8828bcdc9575a04f7716f4254f5ba"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.197870 23002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.209982 23002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:30.212054 23002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:30.212230 23002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:30.212497 23002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:30.212881 23002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:30.212940 23002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.212977 23002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:30.213004 23002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.219460 23002 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:41677
I20260810 04:16:30.219513 23140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:41677 every 8 connection(s)
I20260810 04:16:30.219857 23002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:30.229169 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.220612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:55865 (local address 127.15.216.126:34993)
0810 04:16:30.220682 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.220684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.220876 (+ 192us) server_negotiation.cc:407] Connection header received
0810 04:16:30.221347 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.221348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.221359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.221420 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.222453 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.222966 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.223928 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.224117 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.224167 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.226578 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.226580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.226580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.226587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.227998 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.228380 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.228381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.228382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.228435 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.228658 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.228658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.228659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.228758 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.228808 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.229038 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.229105 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:16:30.229476 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.220539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:55865)
0810 04:16:30.220736 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.220787 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.221267 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.221499 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.221503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.222179 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.222373 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.222378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.223056 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.223057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.223857 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.223859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.224190 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.224190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.224297 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.224844 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.224853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.226482 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.228079 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.228086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.228087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.228312 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.228497 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.228497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.228498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.228603 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.228816 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.228821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.228932 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.229015 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.229136 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260810 04:16:30.229902 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 23002
I20260810 04:16:30.230027 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:30.230077 23141 heartbeater.cc:347] Connected to a master server at 127.15.216.126:34993
I20260810 04:16:30.230183 23141 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:30.230250 16225 external_mini_cluster.cc:1428] Setting key 42a218006349a7ee9f362b947db981cc
I20260810 04:16:30.230370 23141 heartbeater.cc:511] Master 127.15.216.126:34993 requested a full tablet report, sending...
I20260810 04:16:30.230803 22782 ts_manager.cc:194] Registered new tserver with Master: cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677)
I20260810 04:16:30.231179 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:55865
WARNING: no policy specified for oryx/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab.
Entry for principal oryx/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab.
I20260810 04:16:30.261263 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:34993
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab
--principal=oryx/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:30.347123 23146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:30.347411 23146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:30.347465 23146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:30.349107 23146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:30.349153 23146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:30.349181 23146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:30.350739 23146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab
--principal=oryx/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:34993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:30.350981 23146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:30.351215 23146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:30.356475 23154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.356459 23152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.356446 23151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:30.356715 23146 server_base.cc:1034] running on GCE node
I20260810 04:16:30.356974 23146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:30.357236 23146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:30.358412 23146 hybrid_clock.cc:648] HybridClock initialized: now 1786335390358387 us; error 41 us; skew 500 ppm
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:30.361023 23146 init.cc:377] Logged in from keytab as oryx/127.15.216.67@KRBTEST.COM (short username oryx)
I20260810 04:16:30.361469 23146 webserver.cc:460] Webserver started at http://127.15.216.67:37909/ using document root <none> and password file <none>
I20260810 04:16:30.361706 23146 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.361753 23146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.361886 23146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:30.362890 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "010e38b9da5d4e05be07083caea7fdbd"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "230aa2d01d7dd8680808e6995e5a74d0"
server_key_iv: "4f0c0180362615d930b2c61908dd9a49"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.363185 23146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "010e38b9da5d4e05be07083caea7fdbd"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "230aa2d01d7dd8680808e6995e5a74d0"
server_key_iv: "4f0c0180362615d930b2c61908dd9a49"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.364524 23146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:30.365267 23161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.365464 23146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:30.365574 23146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "010e38b9da5d4e05be07083caea7fdbd"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "230aa2d01d7dd8680808e6995e5a74d0"
server_key_iv: "4f0c0180362615d930b2c61908dd9a49"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.365701 23146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.376749 23146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:30.379268 23146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:30.379415 23146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:30.379679 23146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:30.379998 23146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:30.380030 23146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.380082 23146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:30.380108 23146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.386332 23146 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:37721
I20260810 04:16:30.386386 23282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:37721 every 8 connection(s)
I20260810 04:16:30.386685 23146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:30.387858 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 23146
I20260810 04:16:30.387959 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:30.388160 16225 external_mini_cluster.cc:1428] Setting key 092088fa3757f2422222ccb374705efa
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:30.396214 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.387437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:42821 (local address 127.15.216.126:34993)
0810 04:16:30.387482 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.387484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.387708 (+ 224us) server_negotiation.cc:407] Connection header received
0810 04:16:30.388372 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.388374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.388392 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.388471 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.389657 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.390172 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.391157 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.391406 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.391498 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.393812 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.393813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.393814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.393820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.395053 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.395430 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.395431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.395431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.395484 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.395693 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.395694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.395694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.395792 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.395842 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.396090 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.396151 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:16:30.396481 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.387434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:42821)
0810 04:16:30.387591 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.387625 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.388264 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.388569 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.388573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.389343 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.389535 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.389540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.390258 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.390259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.391056 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.391058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.391522 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.391523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.391629 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.392222 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.392230 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.393728 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.395139 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.395145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.395147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.395361 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.395544 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.395545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.395545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.395640 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.395845 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.395850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.395982 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.396063 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.396231 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260810 04:16:30.396903 23283 heartbeater.cc:347] Connected to a master server at 127.15.216.126:34993
I20260810 04:16:30.396996 23283 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:30.397138 23283 heartbeater.cc:511] Master 127.15.216.126:34993 requested a full tablet report, sending...
I20260810 04:16:30.397469 22782 ts_manager.cc:194] Registered new tserver with Master: 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67:37721)
I20260810 04:16:30.397876 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:42821
I20260810 04:16:30.400101 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.216.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.125.keytab.
Entry for principal oryx/127.15.216.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.125.keytab.
I20260810 04:16:30.428059 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:42949
--webserver_interface=127.15.216.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.125.keytab
--principal=oryx/127.15.216.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:34993,127.15.216.125:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:30.514166 23288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:30.514387 23288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:30.514410 23288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:30.515513 23288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:30.515549 23288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:30.515563 23288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:30.515575 23288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:30.515588 23288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:30.517086 23288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:34993,127.15.216.125:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.125.keytab
--principal=oryx/127.15.216.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.216.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
W20260810 04:16:30.517196 23288 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.216.126:34993,127.15.216.125:42949'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 04:16:30.517318 23288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:30.517493 23288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:30.522459 23293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.522473 23294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.522473 23296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:30.522737 23288 server_base.cc:1034] running on GCE node
I20260810 04:16:30.523061 23288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:30.523351 23288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:30.524518 23288 hybrid_clock.cc:648] HybridClock initialized: now 1786335390524508 us; error 31 us; skew 500 ppm
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:30.527230 23288 init.cc:377] Logged in from keytab as oryx/127.15.216.125@KRBTEST.COM (short username oryx)
I20260810 04:16:30.527699 23288 webserver.cc:460] Webserver started at http://127.15.216.125:33489/ using document root <none> and password file <none>
I20260810 04:16:30.527904 23288 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.527943 23288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.528031 23288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:30.529040 23288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/instance:
uuid: "7bed1b57d96347079f8316f3fac0783e"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "4c771dfb34e1f0ec3ff04ec1212ad987"
server_key_iv: "7350d593f0d01e13abad75605a4df551"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.529335 23288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal/instance:
uuid: "7bed1b57d96347079f8316f3fac0783e"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "4c771dfb34e1f0ec3ff04ec1212ad987"
server_key_iv: "7350d593f0d01e13abad75605a4df551"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.530704 23288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:30.531420 23303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.531582 23288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:16:30.531673 23288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
uuid: "7bed1b57d96347079f8316f3fac0783e"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "4c771dfb34e1f0ec3ff04ec1212ad987"
server_key_iv: "7350d593f0d01e13abad75605a4df551"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.531729 23288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.543942 23288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:30.546191 23288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:30.546380 23288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:30.550840 23288 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.125:42949
I20260810 04:16:30.550897 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.125:42949 every 8 connection(s)
I20260810 04:16:30.551262 23288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
I20260810 04:16:30.552268 23359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:30.553330 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 23288
I20260810 04:16:30.554114 23359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.125@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.216.125@KRBTEST.COM
I20260810 04:16:30.563434 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.554850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.15.216.126:34993)
0810 04:16:30.554900 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.554903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.555034 (+ 131us) server_negotiation.cc:407] Connection header received
0810 04:16:30.555139 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.555140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.555152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.555220 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.555777 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.556311 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.557378 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.557715 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.557780 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.560131 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.560132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.560133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.560139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.561879 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.562375 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.562376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.562376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.562447 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.562740 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.562741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.562741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.562896 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.562979 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.563254 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.563332 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:16:30.563736 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.554103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.15.216.125:42949)
0810 04:16:30.554329 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.554338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.554388 (+ 50us) server_negotiation.cc:407] Connection header received
0810 04:16:30.554971 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.554981 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.555042 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.555134 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.555837 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.556819 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.557678 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.558084 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.558155 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.560639 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.560647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.560648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.560654 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.562389 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.562731 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.562734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.562734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.562794 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.563086 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.563087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.563087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.563228 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.563376 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.563472 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.563551 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":115,"threads_started":1}
I20260810 04:16:30.563774 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.554738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39054)
0810 04:16:30.554897 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.554940 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.555035 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.555250 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.555253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.555511 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.555670 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.555674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.556417 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.556419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.557287 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.557290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.557785 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.557786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.557920 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.558559 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.558567 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.560039 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.561961 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.561967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.561969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.562300 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.562534 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.562535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.562535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.562668 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.562972 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.562979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.563119 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.563221 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.563377 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260810 04:16:30.564457 23359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.125@KRBTEST.COM for oryx/127.15.216.125@KRBTEST.COM
I20260810 04:16:30.570533 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.564697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.0.0.1:58564)
0810 04:16:30.564746 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.564757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.564798 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.564910 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.564910 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.565022 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.565053 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.565054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.565654 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.565655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.566155 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.566157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.566499 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.566500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.566558 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.566909 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.566913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.568084 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.569469 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.569470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.569471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.569696 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.569922 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.569922 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.569923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.569978 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.570218 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.570219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.570256 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.570324 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.570426 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:16:30.570583 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.564773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.15.216.125:42949)
0810 04:16:30.564819 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.564821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.564828 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:30.564846 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.564847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.564855 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.564908 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.565095 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.565543 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.566226 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.566408 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.566481 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.568171 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.568172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.568172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.568193 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.569373 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.569788 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.569788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.569789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.569849 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.570049 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.570050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.570051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.570154 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.570226 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.570350 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.570474 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":29}
I20260810 04:16:30.571815 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Bootstrap starting.
I20260810 04:16:30.572664 23359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:30.572988 23359 log.cc:824] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:30.573748 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: No bootstrap required, opened a new log
I20260810 04:16:30.575174 23359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } }
I20260810 04:16:30.575302 23359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:30.575325 23359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bed1b57d96347079f8316f3fac0783e, State: Initialized, Role: FOLLOWER
I20260810 04:16:30.575418 23359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } }
I20260810 04:16:30.575712 23359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:30.575650 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } } }
I20260810 04:16:30.575765 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:16:30.577477 23377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:16:30.577560 23377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:16:30.579429 23288 master_runner.cc:428] Detected that this master 7bed1b57d96347079f8316f3fac0783e is joining an existing cluster
I20260810 04:16:30.579499 23288 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.216.125:42949
I20260810 04:16:30.587823 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.580615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.15.216.126:34993)
0810 04:16:30.580667 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.580670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.580800 (+ 130us) server_negotiation.cc:407] Connection header received
0810 04:16:30.580968 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.580969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.580982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.581037 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.581639 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.582126 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.583257 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.583508 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.583540 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.584780 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.584782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.584783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.584791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.586514 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.587011 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.587013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.587013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.587074 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.587294 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.587295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.587295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.587413 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.587484 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.587644 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.587732 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:16:30.588078 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.580792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.15.216.125:42949)
0810 04:16:30.580830 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.580833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.580957 (+ 124us) server_negotiation.cc:407] Connection header received
0810 04:16:30.581032 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.581033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.581042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.581090 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.581622 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.582341 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.583347 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.583634 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.583706 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.584836 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.584837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.584838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.584845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.586979 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.587337 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.587338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.587338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.587384 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.587551 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.587552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.587552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.587673 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.587735 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.587896 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.587980 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:16:30.588065 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.580526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39068)
0810 04:16:30.580716 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.580730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.580833 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.581087 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.581088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.581410 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.581517 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.581519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.582214 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.582216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.583152 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.583154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.583615 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.583616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.583660 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.584172 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.584176 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.584671 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.586623 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.586624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.586624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.586935 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.587152 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.587153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.587153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.587215 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.587488 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.587489 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.587536 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.587608 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.587716 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260810 04:16:30.588320 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.580760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.0.0.1:58576)
0810 04:16:30.580872 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.580881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.580978 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.581092 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.581092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.581411 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.581517 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.581519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.582481 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.582482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.583266 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.583268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.583720 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.583721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.583799 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.584350 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.584354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.584757 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.587066 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.587067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.587067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.587263 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.587449 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.587449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.587450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.587499 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.587757 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.587758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.587792 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.588091 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.588198 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":4,"thread_start_us":74,"threads_started":1}
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335389, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.126@KRBTEST.COM for oryx/127.15.216.125@KRBTEST.COM
I20260810 04:16:30.597743 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.590467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.0.0.1:58586)
0810 04:16:30.590604 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.590653 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.590723 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.590863 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.590865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.591104 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.591147 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.591149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.592003 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.592003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.592563 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.592565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.593059 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.593060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.593130 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.593549 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.593553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.595041 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.596653 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.596660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.596661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.596856 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.597066 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.597067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.597067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.597122 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.597394 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.597400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.597447 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.597510 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.597664 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260810 04:16:30.597754 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.590651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.15.216.125:42949)
0810 04:16:30.590728 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.590730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.590739 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:30.590783 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.590784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.590794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.590860 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.591218 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.591926 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.592640 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.592983 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.593052 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.595123 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.595125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.595125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.595132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.596570 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.596937 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.596938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.596938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.596997 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.597184 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.597185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.597185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.597327 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.597398 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.597555 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.597662 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 04:16:30.598506 22782 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.216.125:42949: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7bed1b57d96347079f8316f3fac0783e"
member_type: NON_VOTER
last_known_addr {
host: "127.15.216.125"
port: 42949
}
attrs {
promote: true
}
}
dest_uuid: "3fab64e82a934b07aed27d6704f08048"
cas_config_opid_index: -1
I20260810 04:16:30.598824 22782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } }
I20260810 04:16:30.599344 23386 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7bed1b57d96347079f8316f3fac0783e (127.15.216.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } }
I20260810 04:16:30.599694 23331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:30.600502 23386 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.216.125:42949
I20260810 04:16:30.600720 23331 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Refusing update from remote peer 3fab64e82a934b07aed27d6704f08048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 04:16:30.600767 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } }
I20260810 04:16:30.600836 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:30.600885 23288 master.cc:537] Master@127.15.216.125:42949 shutting down...
I20260810 04:16:30.601033 23387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:16:30.601773 22768 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 04:16:30.602205 22768 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 -> Peer 7bed1b57d96347079f8316f3fac0783e (127.15.216.125:42949): Couldn't send request to peer 7bed1b57d96347079f8316f3fac0783e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 04:16:30.603843 23288 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:30.603937 23288 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:30.603976 23288 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: stopping tablet replica
I20260810 04:16:30.616751 23288 master.cc:552] Master@127.15.216.125:42949 shutdown complete.
I20260810 04:16:30.616808 23288 master_runner.cc:305] Clearing existing system tablet
I20260810 04:16:30.616953 23288 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.617007 23288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.617697 23288 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260810 04:16:30.618450 23391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.618568 23288 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.618628 23288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:16:30.618649 23288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
uuid: "7bed1b57d96347079f8316f3fac0783e"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "4c771dfb34e1f0ec3ff04ec1212ad987"
server_key_iv: "7350d593f0d01e13abad75605a4df551"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.619335 23288 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:30.620810 23288 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 04:16:30.620875 23288 log.cc:1196] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 04:16:30.621114 23288 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Deleting consensus metadata
I20260810 04:16:30.621271 23288 master_runner.cc:315] Copying system tablet from 127.15.216.126:34993
I20260810 04:16:30.622080 23288 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.216.126:34993
I20260810 04:16:30.628260 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.622391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39072)
0810 04:16:30.622523 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.622533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.622623 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.622776 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.622777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.622959 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.623015 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.623017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.623707 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.623708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.624266 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.624267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.624628 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.624629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.624698 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.625215 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.625220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.625558 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.627154 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.627155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.627155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.627338 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.627564 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.627565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.627566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.627631 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.627903 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.627904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.627953 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.628040 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.628148 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260810 04:16:30.628247 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.622449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.15.216.126:34993)
0810 04:16:30.622492 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.622494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.622605 (+ 111us) server_negotiation.cc:407] Connection header received
0810 04:16:30.622688 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.622689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.622699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.622771 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.623098 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.623611 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.624362 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.624555 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.624629 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.625686 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.625688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.625688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.625711 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.627068 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.627428 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.627431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.627431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.627497 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.627702 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.627703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.627703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.627817 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.627910 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.628067 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.628146 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:16:30.628587 22806 tablet_copy_service.cc:140] P 3fab64e82a934b07aed27d6704f08048: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7bed1b57d96347079f8316f3fac0783e ({username='oryx', principal='oryx/127.15.216.125@KRBTEST.COM'} at 127.0.0.1:39072)
I20260810 04:16:30.628682 22806 tablet_copy_service.cc:161] P 3fab64e82a934b07aed27d6704f08048: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7bed1b57d96347079f8316f3fac0783e at {username='oryx', principal='oryx/127.15.216.125@KRBTEST.COM'} at 127.0.0.1:39072: session id = 7bed1b57d96347079f8316f3fac0783e-00000000000000000000000000000000
I20260810 04:16:30.629300 22806 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 04:16:30.630023 23288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 04:16:30.630980 23288 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: tablet copy: Starting download of 0 data blocks...
I20260810 04:16:30.631126 23288 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: tablet copy: Starting download of 1 WAL segments...
I20260810 04:16:30.632010 23288 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 04:16:30.632786 22806 tablet_copy_service.cc:342] P 3fab64e82a934b07aed27d6704f08048: Request end of tablet copy session 7bed1b57d96347079f8316f3fac0783e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.216.125@KRBTEST.COM'} at 127.0.0.1:39072
I20260810 04:16:30.632846 22806 tablet_copy_service.cc:433] P 3fab64e82a934b07aed27d6704f08048: ending tablet copy session 7bed1b57d96347079f8316f3fac0783e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7bed1b57d96347079f8316f3fac0783e
I20260810 04:16:30.635476 23288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:30.636669 23399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.636776 23398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:30.636960 23288 server_base.cc:1034] running on GCE node
W20260810 04:16:30.637714 23401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:30.637879 23288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:30.638038 23288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:30.639187 23288 hybrid_clock.cc:648] HybridClock initialized: now 1786335390639160 us; error 45 us; skew 500 ppm
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:30.640543 23288 init.cc:377] Logged in from keytab as oryx/127.15.216.125@KRBTEST.COM (short username oryx)
I20260810 04:16:30.640823 23288 webserver.cc:460] Webserver started at http://127.15.216.125:34509/ using document root <none> and password file <none>
I20260810 04:16:30.640976 23288 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.641017 23288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.641649 23288 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:16:30.642230 23409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.642412 23288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.642472 23288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
uuid: "7bed1b57d96347079f8316f3fac0783e"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "4c771dfb34e1f0ec3ff04ec1212ad987"
server_key_iv: "7350d593f0d01e13abad75605a4df551"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.642532 23288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.680074 23288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:30.681948 23288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:30.685304 23288 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.125:42949
I20260810 04:16:30.685334 23463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.125:42949 every 8 connection(s)
I20260810 04:16:30.685644 23288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
I20260810 04:16:30.686527 23464 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:16:30.686960 23464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Bootstrap starting.
I20260810 04:16:30.689177 23464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:16:30.689390 23464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: Bootstrap complete.
I20260810 04:16:30.689635 23464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } }
I20260810 04:16:30.689733 23464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bed1b57d96347079f8316f3fac0783e, State: Initialized, Role: LEARNER
I20260810 04:16:30.689786 23464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } }
I20260810 04:16:30.689975 23464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:30.689970 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } }
I20260810 04:16:30.690068 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: LEARNER
I20260810 04:16:30.691431 23479 catalog_manager.cc:1259] Loaded cluster ID: c3fc8f39de79412296c255f63da00da1
I20260810 04:16:30.691500 23479 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: loading cluster ID for follower catalog manager: success
I20260810 04:16:30.692416 23479 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: acquiring CA information for follower catalog manager: success
I20260810 04:16:30.692695 23479 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:16:30.722630 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.716107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.15.216.125:42949)
0810 04:16:30.716262 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.716264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.716271 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:30.716294 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.716295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.716303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.716381 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.716943 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.717639 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.718355 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.718561 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.718635 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.719963 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.719965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.719966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.719973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.721451 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.721824 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.721825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.721826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.721893 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.722105 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.722106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.722106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.722218 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.722324 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.722432 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.722518 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.15.216.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.124.keytab.
Entry for principal oryx/127.15.216.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.124.keytab.
I20260810 04:16:30.753755 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:41741
--webserver_interface=127.15.216.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.124.keytab
--principal=oryx/127.15.216.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:30.852028 23483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:30.852272 23483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:30.852319 23483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:30.853477 23483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:30.853541 23483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:30.853566 23483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:30.853593 23483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:30.853647 23483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:30.855145 23483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.124.keytab
--principal=oryx/127.15.216.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.216.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:30.855427 23483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:30.855664 23483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:30.861016 23489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.861027 23488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:30.861024 23491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:30.861232 23483 server_base.cc:1034] running on GCE node
I20260810 04:16:30.861516 23483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:30.861820 23483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:30.862975 23483 hybrid_clock.cc:648] HybridClock initialized: now 1786335390862964 us; error 31 us; skew 500 ppm
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:30.865513 23483 init.cc:377] Logged in from keytab as oryx/127.15.216.124@KRBTEST.COM (short username oryx)
I20260810 04:16:30.865988 23483 webserver.cc:460] Webserver started at http://127.15.216.124:42401/ using document root <none> and password file <none>
I20260810 04:16:30.866212 23483 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.866257 23483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.866428 23483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:30.867620 23483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/instance:
uuid: "f7157b54a5b04db5b6f1d5b76356674b"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "2ceebdcd0d057973a85647c142e22d3c"
server_key_iv: "958d1f575880f0d168af278d5314da1f"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.868129 23483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal/instance:
uuid: "f7157b54a5b04db5b6f1d5b76356674b"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "2ceebdcd0d057973a85647c142e22d3c"
server_key_iv: "958d1f575880f0d168af278d5314da1f"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.869997 23483 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260810 04:16:30.870944 23499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.871129 23483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:30.871235 23483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
uuid: "f7157b54a5b04db5b6f1d5b76356674b"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "2ceebdcd0d057973a85647c142e22d3c"
server_key_iv: "958d1f575880f0d168af278d5314da1f"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.871321 23483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.881263 23483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:30.883152 23483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:30.883314 23483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:30.887517 23483 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.124:41741
I20260810 04:16:30.887571 23554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.124:41741 every 8 connection(s)
I20260810 04:16:30.887938 23483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
I20260810 04:16:30.889024 23555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:30.890496 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 23483
I20260810 04:16:30.890918 23555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.124@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.216.124@KRBTEST.COM
I20260810 04:16:30.900219 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.891738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.15.216.126:34993)
0810 04:16:30.891788 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.891790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.891955 (+ 165us) server_negotiation.cc:407] Connection header received
0810 04:16:30.892100 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.892101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.892114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.892198 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.892812 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.893312 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.894298 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.894494 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.894579 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.897029 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.897031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.897031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.897039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.898759 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.899209 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.899211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.899211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.899274 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.899532 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.899533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.899533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.899660 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.899735 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.900042 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.900124 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:16:30.900578 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.891182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.15.216.124:41741)
0810 04:16:30.891417 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.891427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.891469 (+ 42us) server_negotiation.cc:407] Connection header received
0810 04:16:30.891990 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.891997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.892018 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.892133 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.892739 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.893524 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.894273 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.894602 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.894717 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.897491 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.897503 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.897503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.897509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.899053 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.899420 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.899423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.899423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.899481 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.899694 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.899695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.899696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.899892 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.900039 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.900113 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.900247 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260810 04:16:30.900578 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.891609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39076)
0810 04:16:30.891834 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.891878 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.892009 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.892229 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.892232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.892554 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.892712 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.892716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.893429 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.893430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.894195 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.894198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.894614 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.894616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.894715 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.895383 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.895391 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.896926 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.898860 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.898868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.898870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.899114 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.899352 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.899353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.899353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.899465 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.899737 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.899745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.899899 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.900003 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.900174 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260810 04:16:30.901239 23555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.124@KRBTEST.COM for oryx/127.15.216.125@KRBTEST.COM
I20260810 04:16:30.907888 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.901656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.15.216.125:42949)
0810 04:16:30.901701 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.901703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.901710 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:30.901741 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.901743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.901754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.901825 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.902084 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.902585 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.903255 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.903489 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.903565 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.905264 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.905265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.905266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.905273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.906747 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.907097 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.907098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.907098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.907158 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.907361 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.907362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.907362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.907477 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.907548 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.907718 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.907805 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:16:30.907887 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.901501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.0.0.1:58634)
0810 04:16:30.901554 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.901567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.901628 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.901829 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.901830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.901971 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.902007 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.902009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.902666 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.902668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.903172 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.903173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.903563 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.903563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.903628 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.904087 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.904091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.905170 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.906849 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.906850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.906850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.907023 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.907232 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.907233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.907233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.907293 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.907552 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.907553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.907594 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.907690 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.907788 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 04:16:30.908384 23555 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335390, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.124@KRBTEST.COM for oryx/127.15.216.124@KRBTEST.COM
I20260810 04:16:30.914476 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.908640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.0.0.1:50216)
0810 04:16:30.908694 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.908705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.908748 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.908848 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.908848 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.908953 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.908981 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.908981 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.909640 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.909641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.910101 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.910102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.910475 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.910476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.910511 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.910834 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.910838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.911959 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.913526 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.913526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.913527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.913745 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.913954 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.913955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.913956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.914023 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.914255 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.914255 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.914288 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.914338 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.914418 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:16:30.914538 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.908706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.15.216.124:41741)
0810 04:16:30.908760 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.908762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.908768 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:16:30.908787 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.908787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.908794 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.908845 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.909059 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.909525 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.910194 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.910404 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.910468 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.912030 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.912031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.912032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.912038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.913457 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.913807 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.913808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.913808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.913854 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.914100 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.914101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.914101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.914199 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.914251 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.914355 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.914463 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":37}
I20260810 04:16:30.915987 23555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Bootstrap starting.
I20260810 04:16:30.916818 23555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:30.917114 23555 log.cc:824] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:30.917901 23555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: No bootstrap required, opened a new log
I20260810 04:16:30.919425 23555 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } }
I20260810 04:16:30.919571 23555 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:30.919621 23555 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7157b54a5b04db5b6f1d5b76356674b, State: Initialized, Role: FOLLOWER
I20260810 04:16:30.919759 23555 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } }
I20260810 04:16:30.920123 23561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } } }
I20260810 04:16:30.920198 23555 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:30.920234 23561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:16:30.922032 23573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:16:30.922106 23573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:16:30.924584 23483 master_runner.cc:428] Detected that this master f7157b54a5b04db5b6f1d5b76356674b is joining an existing cluster
I20260810 04:16:30.924638 23483 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.216.124:41741
I20260810 04:16:30.950454 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.925821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.15.216.126:34993)
0810 04:16:30.925868 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.925871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.926033 (+ 162us) server_negotiation.cc:407] Connection header received
0810 04:16:30.926141 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.926142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.926154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.926187 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.926702 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.927506 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.928610 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.928910 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.928949 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.946928 (+ 17979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.946931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.946932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.946942 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.948900 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949426 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.949427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.949428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.949499 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949810 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.949811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.949811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.949972 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.950021 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.950248 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.950342 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:16:30.950466 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.926179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50220 (local address 127.15.216.124:41741)
0810 04:16:30.926257 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.926258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.926265 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:30.926281 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.926282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.926303 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.926360 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.926813 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.927630 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.928809 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.929117 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.929160 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.946968 (+ 17808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.946986 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.946987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.946993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.949200 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949578 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.949579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.949579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.949667 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949876 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.949877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.949878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.950014 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.950059 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.950247 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.950346 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 04:16:30.950850 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.925805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39084)
0810 04:16:30.925969 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.925980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.926061 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.926261 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.926262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.926527 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.926609 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.926611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.927601 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.927602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.928515 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.928517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.929038 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.929039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.929110 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.929619 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.929625 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.946715 (+ 17090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.949005 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.949007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.949309 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.949581 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.949582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.949710 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.950074 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.950075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.950125 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.950597 (+ 472us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.950716 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260810 04:16:30.951038 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.926221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.15.216.125:42949)
0810 04:16:30.926409 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.926411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.926529 (+ 118us) server_negotiation.cc:407] Connection header received
0810 04:16:30.926623 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.926624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.926636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.926668 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.927052 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.927794 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.928751 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.929040 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.929077 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.947520 (+ 18443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.947522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.947522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.947531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.949841 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.950202 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.950204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.950204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.950258 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.950457 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.950457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.950458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.950591 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.950643 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.950869 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.950944 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260810 04:16:30.951385 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.926040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.0.0.1:58650)
0810 04:16:30.926473 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.926483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.926552 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.926738 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.926739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.926911 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.926957 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.926959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.927882 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.927883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.928671 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.928673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.929152 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.929152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.929225 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.929991 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.929995 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.947427 (+ 17432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.949922 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949922 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.949923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.950118 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.950334 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.950335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.950335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.950388 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.950699 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.950700 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.950746 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.951167 (+ 421us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.951280 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":60,"threads_started":1}
I20260810 04:16:30.951752 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.925809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.0.0.1:50220)
0810 04:16:30.926118 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.926130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.926195 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.926390 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.926391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.926626 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.926701 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.926703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.927732 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.927732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.928694 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.928697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.929235 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.929236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.929301 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.946424 (+ 17123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.946431 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.946879 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.949295 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949295 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.949296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.949496 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.949741 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.949742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.949742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.949810 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.950103 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.950103 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.950140 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.951370 (+ 1230us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.951667 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":180,"threads_started":1}
I20260810 04:16:30.952005 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 22852
Aug 10 04:16:30 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335389, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.126@KRBTEST.COM for oryx/127.15.216.124@KRBTEST.COM
I20260810 04:16:30.964794 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:32781
--local_ip_for_outbound_sockets=127.15.216.65
--tserver_master_addrs=127.15.216.126:34993
--webserver_port=37457
--webserver_interface=127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab
--principal=oryx/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:16:30.967751 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.953248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.0.0.1:50224)
0810 04:16:30.958226 (+ 4978us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.958240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.958301 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.959590 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.959591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.959798 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.959853 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.959856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.960817 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.960819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.961814 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.961817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.962334 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.962335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.962415 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.962881 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:30.962887 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:30.964560 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:30.966522 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.966524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.966524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.966755 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.967000 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:30.967001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:30.967001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:30.967075 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:30.967386 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:30.967387 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:30.967439 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.967519 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.967631 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4956}
I20260810 04:16:30.967765 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.953340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.15.216.124:41741)
0810 04:16:30.953378 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.953380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.959446 (+ 6066us) server_negotiation.cc:407] Connection header received
0810 04:16:30.959483 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.959484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.959496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.959555 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.959949 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.960714 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.961922 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.962246 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.962333 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.964655 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:30.964656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:30.964657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:30.964665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:30.966425 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.966848 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.966849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.966850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.966916 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:30.967152 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:30.967152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:30.967153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:30.967308 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:30.967390 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.967577 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.967660 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:16:30.968559 22782 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.216.124:41741: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b"
member_type: NON_VOTER
last_known_addr {
host: "127.15.216.124"
port: 41741
}
attrs {
promote: true
}
}
dest_uuid: "3fab64e82a934b07aed27d6704f08048"
cas_config_opid_index: 5
I20260810 04:16:30.968860 22782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } }
I20260810 04:16:30.969273 23386 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.969575 23529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:30.970309 23386 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.216.124:41741
I20260810 04:16:30.970377 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.970443 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:30.970764 23529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Refusing update from remote peer 3fab64e82a934b07aed27d6704f08048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 04:16:30.971066 23387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:30.971194 23483 master.cc:537] Master@127.15.216.124:41741 shutting down...
W20260810 04:16:30.972010 22769 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 -> Peer f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124:41741): Couldn't send request to peer f7157b54a5b04db5b6f1d5b76356674b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 04:16:30.975096 23483 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:30.975224 23483 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:30.975281 23483 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: stopping tablet replica
I20260810 04:16:30.978859 23483 master.cc:552] Master@127.15.216.124:41741 shutdown complete.
I20260810 04:16:30.978907 23483 master_runner.cc:305] Clearing existing system tablet
I20260810 04:16:30.979043 23483 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:30.979094 23483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:30.979363 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.970741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.15.216.125:42949)
0810 04:16:30.970797 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.970800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.974603 (+ 3803us) server_negotiation.cc:407] Connection header received
0810 04:16:30.974720 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.974721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.974749 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.974804 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:30.975102 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.975978 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.977970 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.978650 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.978707 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.978801 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:30.979233 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 04:16:30.979286 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:16:30.979763 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.970633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.0.0.1:58660)
0810 04:16:30.974532 (+ 3899us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.974546 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.974635 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.974966 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.974967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.974969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:30.975015 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.975017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.976335 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.976336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.977854 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.977858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.978873 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.978875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:30.979027 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:30.979057 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 04:16:30.979639 (+ 582us) client_negotiation.cc:240] Negotiation successful
0810 04:16:30.979689 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3877}
I20260810 04:16:30.980262 23438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 LEARNER]: Refusing update from remote peer 3fab64e82a934b07aed27d6704f08048: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 04:16:30.980495 23387 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 04:16:30.981086 23438 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.982513 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fab64e82a934b07aed27d6704f08048. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.982611 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: LEARNER
I20260810 04:16:30.982649 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: NON_VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: true } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.982702 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: LEARNER
I20260810 04:16:30.982976 23387 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: attempting to promote NON_VOTER 7bed1b57d96347079f8316f3fac0783e to VOTER
I20260810 04:16:30.983196 23387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } }
I20260810 04:16:30.983742 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.983616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.0.0.1:50236)
0810 04:16:30.983662 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.983711 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:16:30.983820 23438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 LEARNER]: Refusing update from remote peer 3fab64e82a934b07aed27d6704f08048: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 04:16:30.984086 23386 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 04:16:30.984350 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.983976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.0.0.1:50242)
0810 04:16:30.984293 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.984321 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":76,"threads_started":1}
W20260810 04:16:30.984404 22769 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 -> Peer f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124:41741): Couldn't send request to peer f7157b54a5b04db5b6f1d5b76356674b. Status: Network error: Client connection negotiation failed: client connection to 127.15.216.124:41741: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 04:16:30.985003 23587 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7bed1b57d96347079f8316f3fac0783e (127.15.216.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.985180 23438 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7bed1b57d96347079f8316f3fac0783e (127.15.216.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.986236 23467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.986321 23467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:16:30.986472 23387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.986557 23387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:30.986655 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:30.986704 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:30.989856 23483 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.000s
I20260810 04:16:30.990576 23589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:30.990710 23483 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:30.990783 23483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:30.990815 23483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
uuid: "f7157b54a5b04db5b6f1d5b76356674b"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "2ceebdcd0d057973a85647c142e22d3c"
server_key_iv: "958d1f575880f0d168af278d5314da1f"
server_key_version: "encryptionkey@0"
I20260810 04:16:30.991683 23483 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:30.993263 23483 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 04:16:30.993346 23483 log.cc:1196] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 04:16:30.993552 23483 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Deleting consensus metadata
I20260810 04:16:30.993690 23483 master_runner.cc:315] Copying system tablet from 127.15.216.126:34993
I20260810 04:16:30.994349 23483 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: tablet copy: Beginning tablet copy session from remote peer at address 127.15.216.126:34993
I20260810 04:16:31.005457 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.997970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.15.216.126:34993)
0810 04:16:30.998022 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:30.998024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:30.998142 (+ 118us) server_negotiation.cc:407] Connection header received
0810 04:16:30.998213 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:30.998213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:30.998225 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:30.998303 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:30.998705 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.999473 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.000668 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.000965 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.001057 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.002251 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.002253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.002253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.002263 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.004073 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.004483 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.004484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.004485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.004560 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.004793 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.004794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.004795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.004958 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.005060 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.005245 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.005354 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:16:31.005486 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:30.997906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39088)
0810 04:16:30.998065 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:30.998077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:30.998144 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:30.998309 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:30.998310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:30.998542 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:30.998623 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:30.998625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:30.999574 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:30.999575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.000580 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.000583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.001048 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.001049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.001128 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.001690 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.001696 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.002147 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.004169 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.004170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.004171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.004401 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.004640 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.004641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.004641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.004713 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.005051 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.005052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.005104 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.005180 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.005373 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260810 04:16:31.006211 22806 tablet_copy_service.cc:140] P 3fab64e82a934b07aed27d6704f08048: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f7157b54a5b04db5b6f1d5b76356674b ({username='oryx', principal='oryx/127.15.216.124@KRBTEST.COM'} at 127.0.0.1:39088)
I20260810 04:16:31.006279 22806 tablet_copy_service.cc:161] P 3fab64e82a934b07aed27d6704f08048: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f7157b54a5b04db5b6f1d5b76356674b at {username='oryx', principal='oryx/127.15.216.124@KRBTEST.COM'} at 127.0.0.1:39088: session id = f7157b54a5b04db5b6f1d5b76356674b-00000000000000000000000000000000
I20260810 04:16:31.006913 22806 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 04:16:31.007483 23483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 04:16:31.008945 23483 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: tablet copy: Starting download of 0 data blocks...
I20260810 04:16:31.009094 23483 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: tablet copy: Starting download of 1 WAL segments...
I20260810 04:16:31.010004 23483 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 04:16:31.010990 22806 tablet_copy_service.cc:342] P 3fab64e82a934b07aed27d6704f08048: Request end of tablet copy session f7157b54a5b04db5b6f1d5b76356674b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.216.124@KRBTEST.COM'} at 127.0.0.1:39088
I20260810 04:16:31.011047 22806 tablet_copy_service.cc:433] P 3fab64e82a934b07aed27d6704f08048: ending tablet copy session f7157b54a5b04db5b6f1d5b76356674b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f7157b54a5b04db5b6f1d5b76356674b
I20260810 04:16:31.014412 23483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:31.015823 23597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:31.018561 23598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:31.018940 23483 server_base.cc:1034] running on GCE node
W20260810 04:16:31.019066 23600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:31.019253 23483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:31.019395 23483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:31.020522 23483 hybrid_clock.cc:648] HybridClock initialized: now 1786335391020532 us; error 45 us; skew 500 ppm
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:31.022197 23483 init.cc:377] Logged in from keytab as oryx/127.15.216.124@KRBTEST.COM (short username oryx)
I20260810 04:16:31.022457 23483 webserver.cc:460] Webserver started at http://127.15.216.124:37709/ using document root <none> and password file <none>
I20260810 04:16:31.022575 23483 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:31.022612 23483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:31.023435 23483 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:31.024055 23607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.024281 23483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:16:31.024329 23483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
uuid: "f7157b54a5b04db5b6f1d5b76356674b"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "2ceebdcd0d057973a85647c142e22d3c"
server_key_iv: "958d1f575880f0d168af278d5314da1f"
server_key_version: "encryptionkey@0"
I20260810 04:16:31.024387 23483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:31.044330 23483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:31.046929 23483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:31.050441 23483 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.124:41741
I20260810 04:16:31.050590 23660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.124:41741 every 8 connection(s)
I20260810 04:16:31.051072 23483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
I20260810 04:16:31.052070 23661 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:16:31.052623 23661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Bootstrap starting.
I20260810 04:16:31.055802 23661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:16:31.056074 23661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Bootstrap complete.
I20260810 04:16:31.056298 23661 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } }
I20260810 04:16:31.056392 23661 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7157b54a5b04db5b6f1d5b76356674b, State: Initialized, Role: LEARNER
I20260810 04:16:31.056430 23661 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } }
I20260810 04:16:31.056617 23661 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:31.056650 23665 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:31.056905 23665 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: This master's current role is: LEARNER
I20260810 04:16:31.058264 23677 catalog_manager.cc:1259] Loaded cluster ID: c3fc8f39de79412296c255f63da00da1
I20260810 04:16:31.058504 23677 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: loading cluster ID for follower catalog manager: success
I20260810 04:16:31.059753 23677 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: acquiring CA information for follower catalog manager: success
I20260810 04:16:31.060107 23677 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 04:16:31.068790 23583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:31.069010 23583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:31.069059 23583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:31.070334 23583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:31.070408 23583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:31.070463 23583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:31.071956 23583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab
--principal=oryx/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:32781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=37457
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:31.072217 23583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:31.072458 23583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:31.077967 23683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:31.077982 23682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:31.078266 23685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:31.078580 23583 server_base.cc:1034] running on GCE node
I20260810 04:16:31.078789 23583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:31.079047 23583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:31.080216 23583 hybrid_clock.cc:648] HybridClock initialized: now 1786335391080202 us; error 34 us; skew 500 ppm
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:31.083222 23583 init.cc:377] Logged in from keytab as oryx/127.15.216.65@KRBTEST.COM (short username oryx)
I20260810 04:16:31.083828 23583 webserver.cc:460] Webserver started at http://127.15.216.65:37457/ using document root <none> and password file <none>
I20260810 04:16:31.084110 23583 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:31.084204 23583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:31.086174 23583 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:31.086930 23692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.087149 23583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:31.087263 23583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "8ca26d5b94ef47049610a78d254d2415"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "5b2fbc480046f2306b9b7febd415104a"
server_key_iv: "7f59fb896ea2661fa4a11994e10ac725"
server_key_version: "encryptionkey@0"
I20260810 04:16:31.087743 23583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:31.121780 23583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:31.124281 23583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:31.124408 23583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:31.124639 23583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:31.124919 23583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:31.124949 23583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.124986 23583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:31.125031 23583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.131645 23583 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:32781
I20260810 04:16:31.131703 23810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:32781 every 8 connection(s)
I20260810 04:16:31.132107 23583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.124@KRBTEST.COM
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.125@KRBTEST.COM
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:31.142553 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 23583
I20260810 04:16:31.142735 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 23002
I20260810 04:16:31.145226 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.133368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:39881 (local address 127.15.216.125:42949)
0810 04:16:31.133410 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.133413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.133783 (+ 370us) server_negotiation.cc:407] Connection header received
0810 04:16:31.134047 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.134048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.134060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.134114 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.134904 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.135650 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.136879 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.137192 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.137262 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.139968 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.139970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.139970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.139978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.142096 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.143899 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.143900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.143901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.143975 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.144374 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.144375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.144375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.144579 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.144635 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.144997 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.145085 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:16:31.145819 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.133193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.15.216.65:39881)
0810 04:16:31.133704 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.133715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.133893 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.134137 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.134141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.134740 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.134809 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.134811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.135754 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.135755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.136758 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.136760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.137270 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.137271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.137330 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.138059 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.138063 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.139861 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.142197 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.142206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.142208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.143787 (+ 1579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.144082 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.144083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.144084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.144278 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.144692 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.144700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.144859 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.145321 (+ 462us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.145535 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":64,"thread_start_us":69,"threads_started":1}
I20260810 04:16:31.146341 23814 heartbeater.cc:347] Connected to a master server at 127.15.216.125:42949
I20260810 04:16:31.146446 23814 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.146641 23814 heartbeater.cc:511] Master 127.15.216.125:42949 requested a full tablet report, sending...
I20260810 04:16:31.147229 23427 ts_manager.cc:194] Registered new tserver with Master: 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65:32781)
I20260810 04:16:31.147780 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.132857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.15.216.65:39663)
0810 04:16:31.133320 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.133329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.133898 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.134251 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.134252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.134561 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.134766 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.134770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.135672 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.135674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.136529 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.136532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.136986 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.136987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.137134 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.137692 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.137703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.139380 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.142258 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.142259 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.142259 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.146657 (+ 4398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.146985 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.146986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.146986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.147090 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.147483 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.147484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.147543 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.147600 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.147712 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":145,"threads_started":1}
I20260810 04:16:31.148175 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.132937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:39663 (local address 127.15.216.124:41741)
0810 04:16:31.133844 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.133846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.133857 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:31.134117 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.134119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.134132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.134209 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.134871 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.135553 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.136628 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.136892 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.136967 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.139482 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.139484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.139484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.139492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.142074 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.146832 (+ 4758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.146834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.146834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.146910 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.147161 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.147162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.147162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.147390 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.147457 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.147981 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.148078 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":58,"threads_started":1}
I20260810 04:16:31.148727 23811 heartbeater.cc:347] Connected to a master server at 127.15.216.124:41741
I20260810 04:16:31.148828 23811 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.148945 23811 heartbeater.cc:511] Master 127.15.216.124:41741 requested a full tablet report, sending...
I20260810 04:16:31.149358 23624 ts_manager.cc:194] Registered new tserver with Master: 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65:32781)
I20260810 04:16:31.149523 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.132849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:58313 (local address 127.15.216.126:34993)
0810 04:16:31.132927 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.132930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.133234 (+ 304us) server_negotiation.cc:407] Connection header received
0810 04:16:31.133968 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.133969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.133981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.134048 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.134887 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.135661 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.136847 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.137120 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.137202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.140494 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.140495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.140496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.140506 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.142291 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.148611 (+ 6320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.148613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.148613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.148704 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.148933 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.148934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.148934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.149068 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.149123 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.149329 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.149424 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:16:31.149821 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.132855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:58313)
0810 04:16:31.133102 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.133147 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.133878 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.134137 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.134141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.134583 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.134767 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.134774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.135754 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.135755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.136758 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.136760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.137203 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.137204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.137278 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.137987 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.137991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.140368 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.142367 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.142368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.142368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.148518 (+ 6150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.148785 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.148786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.148786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.148853 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.149157 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.149158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.149199 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.149645 (+ 446us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.149754 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":43,"thread_start_us":140,"threads_started":1}
I20260810 04:16:31.150120 23813 heartbeater.cc:347] Connected to a master server at 127.15.216.126:34993
I20260810 04:16:31.150166 23813 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.150219 23813 heartbeater.cc:511] Master 127.15.216.126:34993 requested a full tablet report, sending...
I20260810 04:16:31.150461 22782 ts_manager.cc:194] Re-registered known tserver with Master: 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65:32781)
I20260810 04:16:31.150849 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:58313
I20260810 04:16:31.154578 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:41677
--local_ip_for_outbound_sockets=127.15.216.66
--tserver_master_addrs=127.15.216.126:34993
--webserver_port=45493
--webserver_interface=127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab
--principal=oryx/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:31.241571 23820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:31.241783 23820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:31.241807 23820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:31.242996 23820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:31.243036 23820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:31.243093 23820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:31.244565 23820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab
--principal=oryx/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:41677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=45493
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:31.244756 23820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:31.244927 23820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:31.250559 23826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:31.250607 23828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:31.250559 23825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:31.250907 23820 server_base.cc:1034] running on GCE node
I20260810 04:16:31.251118 23820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:31.251336 23820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:31.252482 23820 hybrid_clock.cc:648] HybridClock initialized: now 1786335391252464 us; error 36 us; skew 500 ppm
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:31.255178 23820 init.cc:377] Logged in from keytab as oryx/127.15.216.66@KRBTEST.COM (short username oryx)
I20260810 04:16:31.255594 23820 webserver.cc:460] Webserver started at http://127.15.216.66:45493/ using document root <none> and password file <none>
I20260810 04:16:31.255848 23820 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:31.255951 23820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:31.257788 23820 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:31.258481 23835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.258663 23820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:31.258782 23820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "6888322a49638dc4b51c01be5793abe6"
server_key_iv: "96a8828bcdc9575a04f7716f4254f5ba"
server_key_version: "encryptionkey@0"
I20260810 04:16:31.259150 23820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:31.324874 23820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:31.327188 23820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:31.327379 23820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:31.327653 23820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:31.328011 23820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:31.328083 23820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.328130 23820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:31.328161 23820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.334904 23820 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:41677
I20260810 04:16:31.334961 23954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:41677 every 8 connection(s)
I20260810 04:16:31.335338 23820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:31.341435 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 23820
I20260810 04:16:31.341665 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 23146
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.124@KRBTEST.COM
I20260810 04:16:31.350910 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.336580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:43593 (local address 127.15.216.126:34993)
0810 04:16:31.336637 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.336639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.336711 (+ 72us) server_negotiation.cc:407] Connection header received
0810 04:16:31.337704 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.337705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.337720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.337758 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.338640 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.339525 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.340877 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.341239 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.341323 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.345629 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.345631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.345632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.345642 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.347991 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.348901 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.348903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.348903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.349697 (+ 794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.350048 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.350049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.350050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.350201 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.350264 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.350701 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.350800 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:16:31.350910 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.336115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:52663 (local address 127.15.216.124:41741)
0810 04:16:31.336168 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.336171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.336467 (+ 296us) server_negotiation.cc:407] Connection header received
0810 04:16:31.337704 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.337705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.337720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.337784 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.338640 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.339546 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.340841 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.341095 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.341170 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.347320 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.347321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.347322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.347331 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.349258 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.349744 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.349746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.349746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.349830 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.350053 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.350053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.350054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.350201 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.350261 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.350700 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.350808 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.125@KRBTEST.COM
I20260810 04:16:31.351529 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.336470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:43593)
0810 04:16:31.336641 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.336652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.337544 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.337845 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.337847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.338376 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.338512 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.338521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.339643 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.339644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.340720 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.340723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.341387 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.341389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.341463 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.342673 (+ 1210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.342686 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.345481 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.348096 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.348105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.348108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.348789 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.349794 (+ 1005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.349794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.349795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.349954 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.350319 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.350320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.350498 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.351008 (+ 510us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.351275 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":102,"thread_start_us":71,"threads_started":1}
I20260810 04:16:31.352077 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.336135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.15.216.66:52663)
0810 04:16:31.336325 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.336379 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.337551 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.337869 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.337870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.338377 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.338512 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.338521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.339643 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.339644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.340720 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.340723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.341262 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.341262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.341335 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.345330 (+ 3995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.345335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.347195 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.349363 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.349364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.349365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.349638 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.349914 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.349915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.349915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.349970 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.350301 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.350310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.350498 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.351472 (+ 974us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.351939 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":37,"thread_start_us":107,"threads_started":1}
I20260810 04:16:31.352847 23956 heartbeater.cc:347] Connected to a master server at 127.15.216.126:34993
I20260810 04:16:31.352965 23956 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.353062 23955 heartbeater.cc:347] Connected to a master server at 127.15.216.124:41741
I20260810 04:16:31.353099 23955 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.353197 23956 heartbeater.cc:511] Master 127.15.216.126:34993 requested a full tablet report, sending...
I20260810 04:16:31.353197 23955 heartbeater.cc:511] Master 127.15.216.124:41741 requested a full tablet report, sending...
I20260810 04:16:31.353675 22782 ts_manager.cc:194] Re-registered known tserver with Master: cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677)
I20260810 04:16:31.353686 23624 ts_manager.cc:194] Registered new tserver with Master: cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677)
I20260810 04:16:31.354180 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:43593
I20260810 04:16:31.355141 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.336141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:36247 (local address 127.15.216.125:42949)
0810 04:16:31.336238 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.336241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.336548 (+ 307us) server_negotiation.cc:407] Connection header received
0810 04:16:31.337674 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.337675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.337689 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.337759 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.338620 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.339211 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.340368 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.340597 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.340684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.351657 (+ 10973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.351659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.351660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.351669 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.353839 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.354281 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.354282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.354282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.354345 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.354565 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.354566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.354566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.354719 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.354766 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.354996 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.355076 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 04:16:31.355319 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.336136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.15.216.66:36247)
0810 04:16:31.336447 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.336460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.337513 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.337838 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.337842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.338230 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.338512 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.338517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.339353 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.339355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.340244 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.340247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.340689 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.340691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.340895 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.349899 (+ 9004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.349905 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.351491 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.353941 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.353942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.353942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.354192 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.354426 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.354427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.354427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.354493 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.354831 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.354832 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.354884 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.354959 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.355100 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":52,"thread_start_us":179,"threads_started":1}
I20260810 04:16:31.357983 23957 heartbeater.cc:347] Connected to a master server at 127.15.216.125:42949
I20260810 04:16:31.358052 23957 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.358132 23957 heartbeater.cc:511] Master 127.15.216.125:42949 requested a full tablet report, sending...
I20260810 04:16:31.358471 23427 ts_manager.cc:194] Registered new tserver with Master: cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677)
I20260810 04:16:31.359961 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:37721
--local_ip_for_outbound_sockets=127.15.216.67
--tserver_master_addrs=127.15.216.126:34993
--webserver_port=37909
--webserver_interface=127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab
--principal=oryx/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:44367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:16:31.366812 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.364085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.0.0.1:50258)
0810 04:16:31.364174 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.364187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.364250 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.364466 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.364467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.364469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:31.364533 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.364534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.365185 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.365186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.365996 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.365999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.366565 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.366566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.366628 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.366632 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.366690 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.366740 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 04:16:31.366809 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.364181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.15.216.124:41741)
0810 04:16:31.364227 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.364228 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.364238 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:31.364350 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.364351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.364377 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.364461 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:31.364603 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.365094 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.366081 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.366477 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.366566 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.366639 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.366705 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.366742 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:16:31.367103 23634 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 04:16:31.367290 23665 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fab64e82a934b07aed27d6704f08048. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:31.367441 23665 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: This master's current role is: LEARNER
W20260810 04:16:31.450197 23962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:31.450394 23962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:31.450418 23962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:31.451570 23962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:31.451611 23962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:31.451663 23962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:31.453110 23962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:44367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab
--principal=oryx/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:37721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=37909
--tserver_master_addrs=127.15.216.126:34993,127.15.216.125:42949,127.15.216.124:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:31.453370 23962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:31.453655 23962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:31.459197 23970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:31.459235 23968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:31.459280 23967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:31.459802 23962 server_base.cc:1034] running on GCE node
I20260810 04:16:31.460005 23962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:31.460196 23962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:31.461392 23962 hybrid_clock.cc:648] HybridClock initialized: now 1786335391461367 us; error 36 us; skew 500 ppm
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:31.463956 23962 init.cc:377] Logged in from keytab as oryx/127.15.216.67@KRBTEST.COM (short username oryx)
I20260810 04:16:31.464406 23962 webserver.cc:460] Webserver started at http://127.15.216.67:37909/ using document root <none> and password file <none>
I20260810 04:16:31.464618 23962 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:31.464673 23962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:31.466051 23962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:31.466614 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.466821 23962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:31.466900 23962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "010e38b9da5d4e05be07083caea7fdbd"
format_stamp: "Formatted at 2026-08-10 04:16:30 on dist-test-slave-g350"
server_key: "230aa2d01d7dd8680808e6995e5a74d0"
server_key_iv: "4f0c0180362615d930b2c61908dd9a49"
server_key_version: "encryptionkey@0"
I20260810 04:16:31.467270 23962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:31.479204 23962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:31.481267 23962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:31.481413 23962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:31.481750 23962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:31.482064 23962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:31.482096 23962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.482148 23962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:31.482173 23962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:31.488646 23962 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:37721
I20260810 04:16:31.488696 24092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:37721 every 8 connection(s)
I20260810 04:16:31.489041 23962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:31.495591 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 23962
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.124@KRBTEST.COM
Aug 10 04:16:31 dist-test-slave-g350 krb5kdc[22737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335391, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.125@KRBTEST.COM
I20260810 04:16:31.503592 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.490248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:33113 (local address 127.15.216.125:42949)
0810 04:16:31.490299 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.490301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.490330 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:16:31.491494 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.491496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.491509 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.491566 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.492474 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.493066 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.494687 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.495010 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.495101 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.499894 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.499895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.499896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.499905 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.501845 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.502356 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.502357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.502358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.502442 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.502748 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.502749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.502749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.502897 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.502975 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.503387 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.503468 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:16:31.503777 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.489795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:52031)
0810 04:16:31.490011 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.490059 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.491303 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.491640 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.491644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.492069 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.492298 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.492303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.493337 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.493338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.494506 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.494509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.494986 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.494987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.495088 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.495920 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.495926 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.499622 (+ 3696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.502217 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.502218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.502219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.502449 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.502677 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.502678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.502678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.502739 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.503015 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.503016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.503176 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.503290 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.503510 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":135,"thread_start_us":96,"threads_started":1}
I20260810 04:16:31.504161 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.489795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.15.216.67:37599)
0810 04:16:31.490165 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.490178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.491331 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.491642 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.491646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.492122 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.492300 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.492306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.493269 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.493269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.494407 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.494410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.494948 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.494949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.495092 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.495814 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.495825 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.499316 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.501822 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.501831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.501833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.502278 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.503450 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.503451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.503451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.503515 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.503811 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.503812 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.503860 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.503933 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.504036 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":128,"thread_start_us":93,"threads_started":1}
I20260810 04:16:31.504401 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.490038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:52031 (local address 127.15.216.126:34993)
0810 04:16:31.490113 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.490115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.490143 (+ 28us) server_negotiation.cc:407] Connection header received
0810 04:16:31.491412 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.491414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.491427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.491507 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.492395 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.493225 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.494615 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.494850 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.494884 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.499712 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.499713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.499714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.499723 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.502127 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.502530 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.502531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.502531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.502591 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.502807 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.502808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.502808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.502929 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.502989 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.504220 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.504296 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:16:31.504462 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.490083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.15.216.67:33113)
0810 04:16:31.490261 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.490270 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.491331 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.491639 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.491644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.492091 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:31.492300 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.492306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.493184 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.493185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.494609 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.494611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.495102 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.495103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.495176 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.496046 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:31.496052 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:31.499779 (+ 3727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:31.501946 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.501948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.501948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.502279 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.502514 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:31.502515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:31.502515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:31.502670 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:31.503005 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:31.503013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:31.503179 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.503290 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.504140 (+ 850us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":824,"thread_start_us":59,"threads_started":1}
I20260810 04:16:31.504762 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.489822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:37599 (local address 127.15.216.124:41741)
0810 04:16:31.490109 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.490112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.490231 (+ 119us) server_negotiation.cc:407] Connection header received
0810 04:16:31.491493 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.491495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.491509 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.491564 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.492447 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.493186 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.494510 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.494844 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.494924 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.499472 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.499474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.499475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.499484 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.501697 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.503301 (+ 1604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.503302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.503303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.503368 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.503591 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.503592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.503592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.503725 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.503802 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.504574 (+ 772us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.504660 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260810 04:16:31.505002 24096 heartbeater.cc:347] Connected to a master server at 127.15.216.125:42949
I20260810 04:16:31.505077 24094 heartbeater.cc:347] Connected to a master server at 127.15.216.126:34993
I20260810 04:16:31.505092 24096 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.505148 24094 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.505324 24096 heartbeater.cc:511] Master 127.15.216.125:42949 requested a full tablet report, sending...
I20260810 04:16:31.505671 24093 heartbeater.cc:347] Connected to a master server at 127.15.216.124:41741
I20260810 04:16:31.505716 24093 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:31.505738 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.497468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.15.216.125:42949)
0810 04:16:31.497622 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.497624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.497934 (+ 310us) server_negotiation.cc:407] Connection header received
0810 04:16:31.498041 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.498042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.498054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.498105 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.498807 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.499547 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.500586 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.500829 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.500892 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.502348 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.502349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.502349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.502356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.504260 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.504655 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.504656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.504656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.504725 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.504963 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.504964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.504964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.505101 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.505303 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.505518 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.505653 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260810 04:16:31.505780 24093 heartbeater.cc:511] Master 127.15.216.124:41741 requested a full tablet report, sending...
I20260810 04:16:31.506031 24094 heartbeater.cc:511] Master 127.15.216.126:34993 requested a full tablet report, sending...
I20260810 04:16:31.506225 23624 ts_manager.cc:194] Registered new tserver with Master: 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67:37721)
I20260810 04:16:31.506383 23427 ts_manager.cc:194] Registered new tserver with Master: 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67:37721)
I20260810 04:16:31.506420 22782 ts_manager.cc:194] Re-registered known tserver with Master: 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67:37721)
I20260810 04:16:31.506909 22782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:52031
I20260810 04:16:31.512863 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.496765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.15.216.124:41741)
0810 04:16:31.496895 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.496897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.497946 (+ 1049us) server_negotiation.cc:407] Connection header received
0810 04:16:31.498063 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.498065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.498080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.498126 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.505314 (+ 7188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.506127 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.508050 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.508262 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.508384 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.509807 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.509808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.509809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.509817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.511599 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.511998 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.511999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.511999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.512071 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.512307 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.512308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.512309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.512428 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.512545 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.512658 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.512740 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260810 04:16:31.513262 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.497075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.15.216.126:34993)
0810 04:16:31.497201 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.497203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.497351 (+ 148us) server_negotiation.cc:407] Connection header received
0810 04:16:31.497436 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.497437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.497450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.497511 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.502361 (+ 4850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.503223 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.508230 (+ 5007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.508471 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.508552 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.509947 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.509948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.509948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.509955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.511815 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.512277 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.512278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.512278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.512362 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.512606 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.512606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.512607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.512746 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.512890 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.513043 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.513148 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260810 04:16:31.515170 22782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:31.516970 22782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:31.517720 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: NON_VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: true } } }
I20260810 04:16:31.517838 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:31.519287 23587 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: attempting to promote NON_VOTER f7157b54a5b04db5b6f1d5b76356674b to VOTER
I20260810 04:16:31.519429 23587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:31.519832 23634 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 LEARNER]: Refusing update from remote peer 3fab64e82a934b07aed27d6704f08048: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 04:16:31.520252 23587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 04:16:31.520339 23438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Refusing update from remote peer 3fab64e82a934b07aed27d6704f08048: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 04:16:31.520534 23587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 04:16:31.521260 24113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.521445 23438 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.521407 23634 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.522420 24113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.522516 24113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:31.522580 24113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.522569 23665 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.522646 24113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:31.522670 23665 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:16:31.522710 24113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.522760 24113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:31.522964 24123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3fab64e82a934b07aed27d6704f08048" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:31.523077 24123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:16:31.527724 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.523444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:37721 (local address 127.0.0.1:47470)
0810 04:16:31.523924 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.523934 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.523992 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.524233 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.524234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.524236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:31.524312 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.524315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.525633 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.525635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.526645 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.526647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.527464 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.527465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.527526 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.527530 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.527595 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.527643 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260810 04:16:31.527736 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.523546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.15.216.67:37721)
0810 04:16:31.523751 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.523756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.524018 (+ 262us) server_negotiation.cc:407] Connection header received
0810 04:16:31.524074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.524077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.524097 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.524507 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:31.524575 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.525508 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.526737 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.527367 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.527466 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.527552 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.527613 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.527653 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":127,"threads_started":1}
I20260810 04:16:31.528175 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.523389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41677 (local address 127.0.0.1:44388)
0810 04:16:31.523809 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.523823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.523898 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.524326 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.524327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.524329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:31.524387 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.524389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.525537 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.525538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.526811 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.526814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.527965 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.527966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.528024 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.528028 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.528071 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.528106 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":31}
I20260810 04:16:31.528475 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.523924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.15.216.66:41677)
0810 04:16:31.524130 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.524134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.524145 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:31.524186 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.524188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.524204 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.524530 (+ 326us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:31.524596 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.525437 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.526909 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.527651 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.527720 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.527776 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.528332 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.528394 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260810 04:16:31.528689 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 4df52e9a548d4f6e869375818f217a9d (DEFAULT_TABLE table=test-table [id=4e6bcc955eb84a5db69e95870721f1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:31.529104 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df52e9a548d4f6e869375818f217a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:31.529131 23885 tablet_service.cc:1467] Processing CreateTablet for tablet 4df52e9a548d4f6e869375818f217a9d (DEFAULT_TABLE table=test-table [id=4e6bcc955eb84a5db69e95870721f1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:31.529317 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.524184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.15.216.65:32781)
0810 04:16:31.525645 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.525649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.525659 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:31.525697 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.525698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.525713 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.525801 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:31.526015 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.527043 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.528312 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.528947 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.529037 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.529113 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.529203 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.529240 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1396,"thread_start_us":132,"threads_started":1}
I20260810 04:16:31.529317 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.523534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:32781 (local address 127.0.0.1:59846)
0810 04:16:31.524329 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.524340 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.524396 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.525819 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.525819 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.525822 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:31.525922 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.525925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.527129 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.527130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.528217 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.528220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.529051 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.529052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.529127 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.529132 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.529188 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.529230 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":229,"threads_started":1}
I20260810 04:16:31.529485 23885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df52e9a548d4f6e869375818f217a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:31.530000 23742 tablet_service.cc:1467] Processing CreateTablet for tablet 4df52e9a548d4f6e869375818f217a9d (DEFAULT_TABLE table=test-table [id=4e6bcc955eb84a5db69e95870721f1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:31.530332 23742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df52e9a548d4f6e869375818f217a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:31.533205 24129 tablet_bootstrap.cc:492] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Bootstrap starting.
I20260810 04:16:31.533398 24131 tablet_bootstrap.cc:492] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Bootstrap starting.
I20260810 04:16:31.533752 24130 tablet_bootstrap.cc:492] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Bootstrap starting.
I20260810 04:16:31.534179 24131 tablet_bootstrap.cc:654] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:31.534257 24129 tablet_bootstrap.cc:654] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:31.534539 24130 tablet_bootstrap.cc:654] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:31.534581 24131 log.cc:824] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:31.534605 24129 log.cc:824] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:31.534953 24130 log.cc:824] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:31.535406 24131 tablet_bootstrap.cc:492] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: No bootstrap required, opened a new log
I20260810 04:16:31.535467 24131 ts_tablet_manager.cc:1397] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:16:31.535503 24129 tablet_bootstrap.cc:492] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: No bootstrap required, opened a new log
I20260810 04:16:31.535626 24129 ts_tablet_manager.cc:1397] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:16:31.535817 24130 tablet_bootstrap.cc:492] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: No bootstrap required, opened a new log
I20260810 04:16:31.535884 24130 ts_tablet_manager.cc:1397] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:31.537561 24131 raft_consensus.cc:348] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.537812 24131 raft_consensus.cc:374] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:31.537868 24131 raft_consensus.cc:729] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010e38b9da5d4e05be07083caea7fdbd, State: Initialized, Role: FOLLOWER
I20260810 04:16:31.537815 24129 raft_consensus.cc:348] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.537987 24129 raft_consensus.cc:374] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:31.538007 24131 consensus_queue.cc:260] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.538098 24129 raft_consensus.cc:729] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca26d5b94ef47049610a78d254d2415, State: Initialized, Role: FOLLOWER
I20260810 04:16:31.538316 24131 ts_tablet_manager.cc:1428] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:31.538419 24129 consensus_queue.cc:260] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.538494 24094 heartbeater.cc:503] Master 127.15.216.126:34993 was elected leader, sending a full tablet report...
I20260810 04:16:31.538509 24130 raft_consensus.cc:348] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.538651 24130 raft_consensus.cc:374] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:31.538686 24130 raft_consensus.cc:729] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2c5f98eecd4b1d8271cfa46fd700f6, State: Initialized, Role: FOLLOWER
I20260810 04:16:31.538691 24129 ts_tablet_manager.cc:1428] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:31.538794 24130 consensus_queue.cc:260] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.538883 23813 heartbeater.cc:503] Master 127.15.216.126:34993 was elected leader, sending a full tablet report...
I20260810 04:16:31.539040 24130 ts_tablet_manager.cc:1428] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:16:31.539415 23956 heartbeater.cc:503] Master 127.15.216.126:34993 was elected leader, sending a full tablet report...
I20260810 04:16:31.561780 24135 raft_consensus.cc:482] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:31.561919 24135 raft_consensus.cc:504] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.562333 24135 leader_election.cc:288] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677), 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65:32781)
I20260810 04:16:31.565181 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.562678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:45257 (local address 127.15.216.65:32781)
0810 04:16:31.562725 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.562727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.562736 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:31.562765 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.562765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.562774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.562851 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:31.563046 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.563535 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.564456 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.564816 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.564907 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.564916 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.565071 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.565115 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:16:31.565181 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.562542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:32781 (local address 127.15.216.67:45257)
0810 04:16:31.562624 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.562638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.562695 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.562875 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.562876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.562878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:31.562970 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.562972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.563622 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.563623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.564361 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.564363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.564907 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.564908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.564992 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.564995 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.565065 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.565106 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 04:16:31.565572 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.562542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41677 (local address 127.15.216.67:34381)
0810 04:16:31.562781 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:31.562801 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:31.562883 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:31.563043 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:31.563044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:31.563045 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:31.563125 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.563127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.563833 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.563834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.564642 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:31.564644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.565250 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.565252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:31.565388 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.565393 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:31.565448 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:16:31.565489 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":169}
I20260810 04:16:31.565570 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.562646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:34381 (local address 127.15.216.66:41677)
0810 04:16:31.562699 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.562702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.562874 (+ 172us) server_negotiation.cc:407] Connection header received
0810 04:16:31.562959 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.562960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.562972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.563025 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:31.563210 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.563720 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.564749 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.565149 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.565237 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.565244 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.565455 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.565503 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:16:31.565708 23764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df52e9a548d4f6e869375818f217a9d" candidate_uuid: "010e38b9da5d4e05be07083caea7fdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca26d5b94ef47049610a78d254d2415" is_pre_election: true
I20260810 04:16:31.565887 23764 raft_consensus.cc:2454] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 010e38b9da5d4e05be07083caea7fdbd in term 0.
I20260810 04:16:31.565945 23906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df52e9a548d4f6e869375818f217a9d" candidate_uuid: "010e38b9da5d4e05be07083caea7fdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" is_pre_election: true
I20260810 04:16:31.566119 23906 raft_consensus.cc:2454] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 010e38b9da5d4e05be07083caea7fdbd in term 0.
I20260810 04:16:31.566164 23979 leader_election.cc:302] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010e38b9da5d4e05be07083caea7fdbd, 8ca26d5b94ef47049610a78d254d2415; no voters:
I20260810 04:16:31.566318 24135 raft_consensus.cc:2781] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:31.566392 24135 raft_consensus.cc:482] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:31.566412 24135 raft_consensus.cc:3037] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:31.567121 24135 raft_consensus.cc:504] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.567265 24135 leader_election.cc:288] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [CANDIDATE]: Term 1 election: Requested vote from peers cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677), 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65:32781)
I20260810 04:16:31.567410 23906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df52e9a548d4f6e869375818f217a9d" candidate_uuid: "010e38b9da5d4e05be07083caea7fdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6"
I20260810 04:16:31.567490 23906 raft_consensus.cc:3037] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:31.567517 23764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df52e9a548d4f6e869375818f217a9d" candidate_uuid: "010e38b9da5d4e05be07083caea7fdbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ca26d5b94ef47049610a78d254d2415"
I20260810 04:16:31.567615 23764 raft_consensus.cc:3037] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:31.568360 23906 raft_consensus.cc:2454] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 010e38b9da5d4e05be07083caea7fdbd in term 1.
I20260810 04:16:31.568521 23764 raft_consensus.cc:2454] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 010e38b9da5d4e05be07083caea7fdbd in term 1.
I20260810 04:16:31.568539 23980 leader_election.cc:302] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010e38b9da5d4e05be07083caea7fdbd, cd2c5f98eecd4b1d8271cfa46fd700f6; no voters:
I20260810 04:16:31.568722 24135 raft_consensus.cc:2781] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:31.568872 24135 raft_consensus.cc:686] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 1 LEADER]: Becoming Leader. State: Replica: 010e38b9da5d4e05be07083caea7fdbd, State: Running, Role: LEADER
I20260810 04:16:31.568979 24135 consensus_queue.cc:237] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:31.569868 22783 catalog_manager.cc:5495] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67). New cstate: current_term: 1 leader_uuid: "010e38b9da5d4e05be07083caea7fdbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } health_report { overall_health: UNKNOWN } } }
I20260810 04:16:31.584605 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.582286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.15.216.67:37721)
0810 04:16:31.582359 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.582361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.582388 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:16:31.582450 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.582450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.582467 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.582524 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:31.582642 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.583099 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.583831 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.584069 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.584131 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.584260 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:31.584365 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:31.584434 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.584528 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.584548 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 04:16:31.586225 23958 tablet.cc:2369] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:31.586531 23764 raft_consensus.cc:1261] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 1 FOLLOWER]: Refusing update from remote peer 010e38b9da5d4e05be07083caea7fdbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:31.586532 23906 raft_consensus.cc:1261] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Refusing update from remote peer 010e38b9da5d4e05be07083caea7fdbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:31.586902 24135 consensus_queue.cc:1035] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:31.587035 24138 consensus_queue.cc:1035] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:31.590823 24142 mvcc.cc:205] Tried to move back new op lower bound from 7316829763935449088 to 7316829763867209728. Current Snapshot: MvccSnapshot[applied={T|T < 7316829763935449088}]
I20260810 04:16:31.591464 24143 mvcc.cc:205] Tried to move back new op lower bound from 7316829763935449088 to 7316829763867209728. Current Snapshot: MvccSnapshot[applied={T|T < 7316829763935449088 or (T in {7316829763935449088})}]
I20260810 04:16:31.591741 24141 mvcc.cc:205] Tried to move back new op lower bound from 7316829763935449088 to 7316829763867209728. Current Snapshot: MvccSnapshot[applied={T|T < 7316829763935449088 or (T in {7316829763935449088})}]
I20260810 04:16:31.593333 22783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:31.593437 22783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:31.595388 22783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fab64e82a934b07aed27d6704f08048: Sending DeleteTablet for 3 replicas of tablet 4df52e9a548d4f6e869375818f217a9d
I20260810 04:16:31.595741 23742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df52e9a548d4f6e869375818f217a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:31 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:59846
I20260810 04:16:31.595796 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df52e9a548d4f6e869375818f217a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:31 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:47470
I20260810 04:16:31.595948 23885 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df52e9a548d4f6e869375818f217a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:31 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:44388
I20260810 04:16:31.596192 24152 tablet_replica.cc:330] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: stopping tablet replica
I20260810 04:16:31.596311 24152 raft_consensus.cc:2229] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:31.596432 24152 raft_consensus.cc:2258] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:31.597004 24152 ts_tablet_manager.cc:1905] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:31.597090 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 22750
I20260810 04:16:31.598949 24152 ts_tablet_manager.cc:1918] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:31.599038 24152 log.cc:1196] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/4df52e9a548d4f6e869375818f217a9d
I20260810 04:16:31.599359 24152 ts_tablet_manager.cc:1939] T 4df52e9a548d4f6e869375818f217a9d P cd2c5f98eecd4b1d8271cfa46fd700f6: Deleting consensus metadata
I20260810 04:16:31.601733 24151 tablet_replica.cc:330] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: stopping tablet replica
I20260810 04:16:31.601861 24151 raft_consensus.cc:2229] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:31.602005 24151 raft_consensus.cc:2258] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:31.602525 24151 ts_tablet_manager.cc:1905] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:31.604944 24151 ts_tablet_manager.cc:1918] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:31.605027 24151 log.cc:1196] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/4df52e9a548d4f6e869375818f217a9d
I20260810 04:16:31.605377 24151 ts_tablet_manager.cc:1939] T 4df52e9a548d4f6e869375818f217a9d P 010e38b9da5d4e05be07083caea7fdbd: Deleting consensus metadata
I20260810 04:16:31.606361 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.596297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.15.216.124:41741)
0810 04:16:31.596339 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.596341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.596517 (+ 176us) server_negotiation.cc:407] Connection header received
0810 04:16:31.596567 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.596568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.596582 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.596615 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.597694 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.598681 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.600162 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.600536 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.600620 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.602545 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.602547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.602547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.602555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.604476 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.604907 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.604908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.604909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.604970 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.605293 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.605294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.605295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.605689 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.605778 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.606141 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.606245 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 04:16:31.608512 23980 connection.cc:535] server connection from 127.0.0.1:47470 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 04:16:31.608541 23837 connection.cc:535] server connection from 127.0.0.1:44388 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:16:31.609367 24150 tablet_replica.cc:330] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: stopping tablet replica
I20260810 04:16:31.609494 24150 raft_consensus.cc:2229] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:31.609633 24150 raft_consensus.cc:2258] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:31.610172 24150 ts_tablet_manager.cc:1905] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:31.612317 24150 ts_tablet_manager.cc:1918] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:31.612385 24150 log.cc:1196] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/4df52e9a548d4f6e869375818f217a9d
I20260810 04:16:31.612699 24150 ts_tablet_manager.cc:1939] T 4df52e9a548d4f6e869375818f217a9d P 8ca26d5b94ef47049610a78d254d2415: Deleting consensus metadata
W20260810 04:16:31.613056 23696 connection.cc:441] server connection from 127.0.0.1:59846 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59846 (request call id 1) could send its response
I20260810 04:16:31.618558 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.610259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.15.216.124:41741)
0810 04:16:31.610299 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.610301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.610827 (+ 526us) server_negotiation.cc:407] Connection header received
0810 04:16:31.610929 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.610931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.610943 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.611012 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.611745 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.612596 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.614065 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.614278 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.614362 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.615727 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.615728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.615729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.615736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.617205 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.617620 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.617621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.617622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.617686 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.618002 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.618003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.618004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.618135 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.618252 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.618357 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.618438 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:16:31.621886 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:31.610490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.15.216.125:42949)
0810 04:16:31.610534 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:31.610536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:31.613809 (+ 3273us) server_negotiation.cc:407] Connection header received
0810 04:16:31.613854 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:31.613855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:31.613900 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:31.613966 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:31.614757 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.615708 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.616876 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:31.617238 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:31.617283 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:31.619095 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:31.619097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:31.619097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:31.619103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:31.620513 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.620920 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.620922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.620922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.620972 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:31.621186 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:31.621187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:31.621187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:31.621285 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:31.621354 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:31.621503 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:16:31.621569 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:16:32.591166 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:32.590797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:48749)
0810 04:16:32.590935 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:32.591093 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
W20260810 04:16:32.591557 23813 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:34993 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:32.591971 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:32.591239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:39491)
0810 04:16:32.591836 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:32.591918 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":91,"threads_started":1}
W20260810 04:16:32.592223 23956 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:34993 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:32.592489 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:32.591893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:50911)
0810 04:16:32.592274 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:32.592401 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":55,"threads_started":1}
W20260810 04:16:32.592687 24094 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:34993 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:33.095769 24165 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fab64e82a934b07aed27d6704f08048)
I20260810 04:16:33.095978 24166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3fab64e82a934b07aed27d6704f08048)
I20260810 04:16:33.096009 24165 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:33.096227 24166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:33.096329 24165 leader_election.cc:288] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993), 7bed1b57d96347079f8316f3fac0783e (127.15.216.125:42949)
I20260810 04:16:33.096556 24166 leader_election.cc:288] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993), f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124:41741)
I20260810 04:16:33.096930 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.096544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39634)
0810 04:16:33.096814 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.096880 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":81,"threads_started":1}
I20260810 04:16:33.097040 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.096681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39636)
0810 04:16:33.096918 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.096986 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":59,"threads_started":1}
W20260810 04:16:33.097152 23609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
W20260810 04:16:33.097240 23411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:33.097826 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.097691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39650)
0810 04:16:33.097755 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.097794 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 04:16:33.098057 23609 leader_election.cc:334] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993): Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:33.098253 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.097863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39658)
0810 04:16:33.098163 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.098195 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 04:16:33.098336 23411 leader_election.cc:334] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993): Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:33.100188 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.096782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:42949 (local address 127.0.0.1:58694)
0810 04:16:33.096974 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.096986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:33.097058 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:33.097252 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:33.097253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:33.097256 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:33.097320 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:33.097323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.098274 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.098275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:33.099266 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:33.099269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.099884 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.099885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:33.099953 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:33.099958 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:33.100040 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:16:33.100100 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":58,"threads_started":1}
I20260810 04:16:33.100198 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.096824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.15.216.125:42949)
0810 04:16:33.096896 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:33.096898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:33.097087 (+ 189us) server_negotiation.cc:407] Connection header received
0810 04:16:33.097120 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:33.097121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:33.097130 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:33.097192 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:33.097421 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.098194 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.099385 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.099799 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.099857 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:33.099864 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:33.100055 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 04:16:33.100099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"thread_start_us":33,"threads_started":1}
I20260810 04:16:33.100363 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.096886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.15.216.124:41741)
0810 04:16:33.097248 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:33.097250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:33.097260 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:33.097289 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:33.097290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:33.097302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:33.097396 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:33.097649 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.098466 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.099504 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.099973 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.100050 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:33.100058 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:33.100207 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:16:33.100264 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":42,"threads_started":1}
I20260810 04:16:33.100569 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.096697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:41741 (local address 127.0.0.1:50292)
0810 04:16:33.096943 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.096952 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:33.097016 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:33.097461 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:33.097463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:33.097465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:33.097520 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:33.097522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.098574 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.098575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:33.099400 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:33.099401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:33.100058 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:33.100060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:33.100114 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:33.100117 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:33.100176 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:16:33.100213 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":49,"threads_started":1}
I20260810 04:16:33.100755 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7157b54a5b04db5b6f1d5b76356674b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bed1b57d96347079f8316f3fac0783e" is_pre_election: true
I20260810 04:16:33.100885 23438 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7157b54a5b04db5b6f1d5b76356674b in term 1.
I20260810 04:16:33.100883 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bed1b57d96347079f8316f3fac0783e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7157b54a5b04db5b6f1d5b76356674b" is_pre_election: true
I20260810 04:16:33.100986 23634 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bed1b57d96347079f8316f3fac0783e in term 1.
I20260810 04:16:33.101037 23609 leader_election.cc:302] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bed1b57d96347079f8316f3fac0783e, f7157b54a5b04db5b6f1d5b76356674b; no voters: 3fab64e82a934b07aed27d6704f08048
I20260810 04:16:33.101173 24165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 04:16:33.101199 23412 leader_election.cc:302] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bed1b57d96347079f8316f3fac0783e, f7157b54a5b04db5b6f1d5b76356674b; no voters: 3fab64e82a934b07aed27d6704f08048
I20260810 04:16:33.101262 24165 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fab64e82a934b07aed27d6704f08048)
I20260810 04:16:33.101306 24165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:16:33.101362 24166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 04:16:33.101400 24166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3fab64e82a934b07aed27d6704f08048)
I20260810 04:16:33.101421 24166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:16:33.102351 24166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:33.102391 24165 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:33.102511 24166 leader_election.cc:288] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [CANDIDATE]: Term 2 election: Requested vote from peers 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993), f7157b54a5b04db5b6f1d5b76356674b (127.15.216.124:41741)
I20260810 04:16:33.102576 24165 leader_election.cc:288] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 2 election: Requested vote from peers 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993), 7bed1b57d96347079f8316f3fac0783e (127.15.216.125:42949)
I20260810 04:16:33.102751 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.102639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39682)
0810 04:16:33.102685 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.102721 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 04:16:33.102836 23634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bed1b57d96347079f8316f3fac0783e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7157b54a5b04db5b6f1d5b76356674b"
I20260810 04:16:33.102922 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.102819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39694)
0810 04:16:33.102861 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.102896 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260810 04:16:33.102929 23634 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7bed1b57d96347079f8316f3fac0783e in current term 2: Already voted for candidate f7157b54a5b04db5b6f1d5b76356674b in this term.
I20260810 04:16:33.102964 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7157b54a5b04db5b6f1d5b76356674b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bed1b57d96347079f8316f3fac0783e"
I20260810 04:16:33.103147 23438 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f7157b54a5b04db5b6f1d5b76356674b in current term 2: Already voted for candidate 7bed1b57d96347079f8316f3fac0783e in this term.
I20260810 04:16:33.103391 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.103290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39720)
0810 04:16:33.103330 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.103371 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260810 04:16:33.103401 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.103252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39708)
0810 04:16:33.103350 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.103376 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 04:16:33.103467 23609 leader_election.cc:334] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993): Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
W20260810 04:16:33.103490 23411 leader_election.cc:334] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993): Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:33.103513 23609 leader_election.cc:302] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7157b54a5b04db5b6f1d5b76356674b; no voters: 3fab64e82a934b07aed27d6704f08048, 7bed1b57d96347079f8316f3fac0783e
I20260810 04:16:33.103538 23411 leader_election.cc:302] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bed1b57d96347079f8316f3fac0783e; no voters: 3fab64e82a934b07aed27d6704f08048, f7157b54a5b04db5b6f1d5b76356674b
I20260810 04:16:33.103596 24165 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 04:16:33.103669 24166 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 04:16:33.592473 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.591989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:42093)
0810 04:16:33.592212 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.592267 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260810 04:16:33.592921 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.592601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:55503)
0810 04:16:33.592817 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.592871 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":68,"threads_started":1}
I20260810 04:16:33.593252 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:33.593052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:56467)
0810 04:16:33.593162 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:33.593205 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260810 04:16:34.593726 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.593426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:45517)
0810 04:16:34.593626 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.593669 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260810 04:16:34.594228 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.594001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:43089)
0810 04:16:34.594121 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.594172 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260810 04:16:34.594281 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.593842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:50479)
0810 04:16:34.594176 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.594216 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":70,"threads_started":1}
I20260810 04:16:34.758793 24181 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:34.758898 24181 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:34.759076 24181 leader_election.cc:288] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993), 7bed1b57d96347079f8316f3fac0783e (127.15.216.125:42949)
I20260810 04:16:34.759461 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.759221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39976)
0810 04:16:34.759369 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.759419 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":49,"threads_started":1}
I20260810 04:16:34.759505 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7157b54a5b04db5b6f1d5b76356674b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bed1b57d96347079f8316f3fac0783e" is_pre_election: true
I20260810 04:16:34.759632 23438 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7157b54a5b04db5b6f1d5b76356674b in term 2.
I20260810 04:16:34.759889 23609 leader_election.cc:302] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bed1b57d96347079f8316f3fac0783e, f7157b54a5b04db5b6f1d5b76356674b; no voters:
I20260810 04:16:34.760015 24181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 04:16:34.760080 24181 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:34.760124 24181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:16:34.760169 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.759843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39978)
0810 04:16:34.759939 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.760077 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260810 04:16:34.760353 23609 leader_election.cc:334] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993): Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:34.761010 24181 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:34.761153 24181 leader_election.cc:288] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 3 election: Requested vote from peers 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993), 7bed1b57d96347079f8316f3fac0783e (127.15.216.125:42949)
I20260810 04:16:34.761396 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.761283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39980)
0810 04:16:34.761337 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.761369 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:16:34.761500 23438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7157b54a5b04db5b6f1d5b76356674b" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bed1b57d96347079f8316f3fac0783e"
I20260810 04:16:34.761634 23438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:16:34.761736 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.761632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:39992)
0810 04:16:34.761692 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.761716 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 04:16:34.761822 23609 leader_election.cc:334] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993): Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
I20260810 04:16:34.762400 23438 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7157b54a5b04db5b6f1d5b76356674b in term 3.
I20260810 04:16:34.762639 23609 leader_election.cc:302] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bed1b57d96347079f8316f3fac0783e, f7157b54a5b04db5b6f1d5b76356674b; no voters: 3fab64e82a934b07aed27d6704f08048
I20260810 04:16:34.762732 24181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 3 FOLLOWER]: Leader election won for term 3
I20260810 04:16:34.762849 24181 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [term 3 LEADER]: Becoming Leader. State: Replica: f7157b54a5b04db5b6f1d5b76356674b, State: Running, Role: LEADER
I20260810 04:16:34.762964 24181 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } }
I20260810 04:16:34.763350 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7157b54a5b04db5b6f1d5b76356674b. Latest consensus state: current_term: 3 leader_uuid: "f7157b54a5b04db5b6f1d5b76356674b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:34.763434 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:34.763607 24186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:34.764107 24186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:34.764217 24186 catalog_manager.cc:1259] Loaded cluster ID: c3fc8f39de79412296c255f63da00da1
I20260810 04:16:34.764252 24186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:34.764438 24186 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:34.764537 24186 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Loaded TSK: 0
I20260810 04:16:34.764680 24186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:16:34.790844 23624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:34.792614 23624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:34.793831 23438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [term 3 FOLLOWER]: Refusing update from remote peer f7157b54a5b04db5b6f1d5b76356674b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 04:16:34.793882 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.793753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:40014)
0810 04:16:34.793810 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.793849 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 04:16:34.794162 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.794093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.0.0.1:40024)
0810 04:16:34.794124 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.794146 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260810 04:16:34.794198 24184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 04:16:34.794343 23609 consensus_peers.cc:487] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b -> Peer 3fab64e82a934b07aed27d6704f08048 (127.15.216.126:34993): Couldn't send request to peer 3fab64e82a934b07aed27d6704f08048. Status: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 04:16:34.794905 24188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7157b54a5b04db5b6f1d5b76356674b. Latest consensus state: current_term: 3 leader_uuid: "f7157b54a5b04db5b6f1d5b76356674b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:34.795032 24188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:16:34.795571 24184 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f7157b54a5b04db5b6f1d5b76356674b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:34.795666 24184 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:34.796202 24189 mvcc.cc:205] Tried to move back new op lower bound from 7316829777072967680 to 7316829776949805056. Current Snapshot: MvccSnapshot[applied={T|T < 7316829777072967680 or (T in {7316829777072967680})}]
I20260810 04:16:34.796624 24188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f7157b54a5b04db5b6f1d5b76356674b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "3fab64e82a934b07aed27d6704f08048" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 34993 } } peers { permanent_uuid: "7bed1b57d96347079f8316f3fac0783e" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 42949 } attrs { promote: false } } peers { permanent_uuid: "f7157b54a5b04db5b6f1d5b76356674b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 41741 } attrs { promote: false } } }
I20260810 04:16:34.796720 24188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bed1b57d96347079f8316f3fac0783e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:16:34.800618 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.797408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:37721 (local address 127.0.0.1:47484)
0810 04:16:34.797452 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.797462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:34.797509 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:34.797917 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:34.797918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:34.797919 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:34.797979 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:34.797980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.798660 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.798661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:34.799495 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:34.799497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.800382 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.800383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:34.800439 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:34.800442 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:34.800510 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:16:34.800553 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 04:16:34.800629 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.797471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.15.216.67:37721)
0810 04:16:34.797781 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:34.797783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:34.797793 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:34.797823 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:34.797824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:34.797835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:34.797916 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:34.798045 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.798568 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.799583 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.800235 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.800344 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:34.800359 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:34.800516 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 04:16:34.800562 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":56,"threads_started":1}
I20260810 04:16:34.800920 24027 tablet_service.cc:1467] Processing CreateTablet for tablet ba78483b8dd9427aa6d71699208c073a (DEFAULT_TABLE table=test-table [id=ffcce051c9b048c4885a27a85dd2b889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:34.800990 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.797659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.15.216.66:41677)
0810 04:16:34.797791 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:34.797793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:34.797802 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:34.797827 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:34.797828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:34.797838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:34.797916 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:34.798111 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.798622 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.800021 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.800615 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.800663 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:34.800673 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:34.800874 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 04:16:34.800916 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260810 04:16:34.801087 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba78483b8dd9427aa6d71699208c073a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:34.801200 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.797451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41677 (local address 127.0.0.1:44402)
0810 04:16:34.797614 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.797624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:34.797679 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:34.797972 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:34.797973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:34.797975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:34.798032 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:34.798033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.798719 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.798720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:34.799921 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:34.799923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.800736 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.800737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:34.800798 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:34.800801 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:34.801079 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 04:16:34.801133 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260810 04:16:34.801462 23885 tablet_service.cc:1467] Processing CreateTablet for tablet ba78483b8dd9427aa6d71699208c073a (DEFAULT_TABLE table=test-table [id=ffcce051c9b048c4885a27a85dd2b889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:34.801574 23885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba78483b8dd9427aa6d71699208c073a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:34.802045 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.797533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:32781 (local address 127.0.0.1:59856)
0810 04:16:34.797988 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.797998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:34.798041 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:34.798917 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:34.798918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:34.798921 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:34.798955 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:34.798956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.799916 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.799917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:34.801008 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:34.801010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.801813 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.801814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:34.801871 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:34.801876 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:34.801939 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:16:34.801977 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":109,"threads_started":1}
I20260810 04:16:34.802107 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.797739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.15.216.65:32781)
0810 04:16:34.798778 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:34.798780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:34.798787 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:34.798813 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:34.798814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:34.798824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:34.798897 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:34.799040 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.799797 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.801144 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:34.801720 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:34.801799 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:34.801811 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:34.801960 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:16:34.802011 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":63,"threads_started":1}
I20260810 04:16:34.802326 23742 tablet_service.cc:1467] Processing CreateTablet for tablet ba78483b8dd9427aa6d71699208c073a (DEFAULT_TABLE table=test-table [id=ffcce051c9b048c4885a27a85dd2b889]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:34.802451 23742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba78483b8dd9427aa6d71699208c073a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:34.802676 24198 tablet_bootstrap.cc:492] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Bootstrap starting.
I20260810 04:16:34.803102 24198 tablet_bootstrap.cc:654] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:34.803393 24199 tablet_bootstrap.cc:492] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6: Bootstrap starting.
I20260810 04:16:34.803835 24199 tablet_bootstrap.cc:654] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:34.804318 24201 tablet_bootstrap.cc:492] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415: Bootstrap starting.
I20260810 04:16:34.804474 24198 tablet_bootstrap.cc:492] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: No bootstrap required, opened a new log
I20260810 04:16:34.804543 24198 ts_tablet_manager.cc:1397] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:34.804555 24199 tablet_bootstrap.cc:492] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6: No bootstrap required, opened a new log
I20260810 04:16:34.804600 24199 ts_tablet_manager.cc:1397] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:34.804741 24201 tablet_bootstrap.cc:654] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:34.804716 24199 raft_consensus.cc:348] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:34.804718 24198 raft_consensus.cc:348] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:34.804781 24198 raft_consensus.cc:374] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:34.804780 24199 raft_consensus.cc:374] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:34.804803 24198 raft_consensus.cc:729] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010e38b9da5d4e05be07083caea7fdbd, State: Initialized, Role: FOLLOWER
I20260810 04:16:34.804822 24199 raft_consensus.cc:729] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2c5f98eecd4b1d8271cfa46fd700f6, State: Initialized, Role: FOLLOWER
I20260810 04:16:34.804857 24198 consensus_queue.cc:260] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:34.804868 24199 consensus_queue.cc:260] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:34.805048 24198 ts_tablet_manager.cc:1428] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:34.805058 24199 ts_tablet_manager.cc:1428] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:34.805395 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.805281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:46985)
0810 04:16:34.805327 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.805363 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 04:16:34.805543 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.805300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:46587)
0810 04:16:34.805482 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.805512 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260810 04:16:34.805742 23955 heartbeater.cc:503] Master 127.15.216.124:41741 was elected leader, sending a full tablet report...
I20260810 04:16:34.806059 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.805955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:38327)
0810 04:16:34.805997 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.806028 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 04:16:34.806218 24093 heartbeater.cc:503] Master 127.15.216.124:41741 was elected leader, sending a full tablet report...
I20260810 04:16:34.806345 24201 tablet_bootstrap.cc:492] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415: No bootstrap required, opened a new log
I20260810 04:16:34.806399 24201 ts_tablet_manager.cc:1397] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:34.806504 24201 raft_consensus.cc:348] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:34.806560 24201 raft_consensus.cc:374] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:34.806573 24201 raft_consensus.cc:729] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca26d5b94ef47049610a78d254d2415, State: Initialized, Role: FOLLOWER
I20260810 04:16:34.806630 24201 consensus_queue.cc:260] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:34.806797 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.806605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:57957)
0810 04:16:34.806735 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.806763 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 04:16:34.806820 24201 ts_tablet_manager.cc:1428] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:34.807252 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:34.807110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:57611)
0810 04:16:34.807179 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:34.807207 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 04:16:34.885094 23815 tablet.cc:2369] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:16:34.992213 24097 tablet.cc:2369] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:35.043157 24206 raft_consensus.cc:482] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:35.043397 24206 raft_consensus.cc:504] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:35.043901 24206 leader_election.cc:288] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67:37721), cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677)
I20260810 04:16:35.047140 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.044394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:45821 (local address 127.15.216.67:37721)
0810 04:16:35.044472 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:35.044473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:35.044481 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:35.044507 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:35.044507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:35.044515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:35.044577 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:35.044767 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.045276 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.046411 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.046772 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.046870 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:35.046880 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:35.047030 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 04:16:35.047066 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 04:16:35.047150 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.044113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:37721 (local address 127.15.216.65:45821)
0810 04:16:35.044210 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.044225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:35.044285 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:35.044600 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:35.044602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:35.044604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:35.044691 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:35.044693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.045369 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.045371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:35.046265 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:35.046267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.046863 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.046864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:35.046935 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:35.046938 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:35.047010 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:16:35.047051 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":24}
I20260810 04:16:35.047183 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.044300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:57529 (local address 127.15.216.66:41677)
0810 04:16:35.044362 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:35.044364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:35.044479 (+ 115us) server_negotiation.cc:407] Connection header received
0810 04:16:35.044566 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:35.044567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:35.044576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:35.044680 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:35.044836 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.045332 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.046387 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.046778 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.046876 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:35.046887 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:35.047055 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 04:16:35.047100 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 04:16:35.047251 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.044113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41677 (local address 127.15.216.65:57529)
0810 04:16:35.044378 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.044390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:35.044494 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:35.044691 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:35.044692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:35.044694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:35.044759 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:35.044760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.045425 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.045426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:35.046282 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:35.046284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.046896 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.046898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:35.046973 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:35.046976 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:35.047047 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:16:35.047087 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":43,"thread_start_us":129,"threads_started":1}
I20260810 04:16:35.047515 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78483b8dd9427aa6d71699208c073a" candidate_uuid: "8ca26d5b94ef47049610a78d254d2415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010e38b9da5d4e05be07083caea7fdbd" is_pre_election: true
I20260810 04:16:35.047644 24047 raft_consensus.cc:2454] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca26d5b94ef47049610a78d254d2415 in term 0.
I20260810 04:16:35.047904 23693 leader_election.cc:302] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010e38b9da5d4e05be07083caea7fdbd, 8ca26d5b94ef47049610a78d254d2415; no voters:
I20260810 04:16:35.047875 23906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78483b8dd9427aa6d71699208c073a" candidate_uuid: "8ca26d5b94ef47049610a78d254d2415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" is_pre_election: true
I20260810 04:16:35.047987 23906 raft_consensus.cc:2454] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca26d5b94ef47049610a78d254d2415 in term 0.
I20260810 04:16:35.048058 24206 raft_consensus.cc:2781] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:35.048132 24206 raft_consensus.cc:482] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:35.048156 24206 raft_consensus.cc:3037] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:35.048928 24206 raft_consensus.cc:504] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:35.049083 24206 leader_election.cc:288] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [CANDIDATE]: Term 1 election: Requested vote from peers 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67:37721), cd2c5f98eecd4b1d8271cfa46fd700f6 (127.15.216.66:41677)
I20260810 04:16:35.049242 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78483b8dd9427aa6d71699208c073a" candidate_uuid: "8ca26d5b94ef47049610a78d254d2415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "010e38b9da5d4e05be07083caea7fdbd"
I20260810 04:16:35.049268 23906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba78483b8dd9427aa6d71699208c073a" candidate_uuid: "8ca26d5b94ef47049610a78d254d2415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6"
I20260810 04:16:35.049355 23906 raft_consensus.cc:3037] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:35.049330 24047 raft_consensus.cc:3037] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:35.050031 23906 raft_consensus.cc:2454] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca26d5b94ef47049610a78d254d2415 in term 1.
I20260810 04:16:35.050207 24047 raft_consensus.cc:2454] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca26d5b94ef47049610a78d254d2415 in term 1.
I20260810 04:16:35.050241 23695 leader_election.cc:302] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca26d5b94ef47049610a78d254d2415, cd2c5f98eecd4b1d8271cfa46fd700f6; no voters:
I20260810 04:16:35.050354 24206 raft_consensus.cc:2781] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:35.050560 24206 raft_consensus.cc:686] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca26d5b94ef47049610a78d254d2415, State: Running, Role: LEADER
I20260810 04:16:35.050662 23811 heartbeater.cc:503] Master 127.15.216.124:41741 was elected leader, sending a full tablet report...
I20260810 04:16:35.050662 24206 consensus_queue.cc:237] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } }
I20260810 04:16:35.051162 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.050798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:38547)
0810 04:16:35.051101 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.051135 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260810 04:16:35.051615 23623 catalog_manager.cc:5495] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65). New cstate: current_term: 1 leader_uuid: "8ca26d5b94ef47049610a78d254d2415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ca26d5b94ef47049610a78d254d2415" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32781 } health_report { overall_health: HEALTHY } } }
I20260810 04:16:35.085707 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.083269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.15.216.65:32781)
0810 04:16:35.083350 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:35.083353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:35.083367 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:16:35.083405 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:35.083406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:35.083424 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:35.083506 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:35.083624 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.084102 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.084850 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:35.085070 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:35.085145 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:35.085337 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:35.085443 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:35.085525 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:35.085568 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:16:35.085614 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:16:35.086865 23906 raft_consensus.cc:1261] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Refusing update from remote peer 8ca26d5b94ef47049610a78d254d2415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:35.086977 24047 raft_consensus.cc:1261] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 1 FOLLOWER]: Refusing update from remote peer 8ca26d5b94ef47049610a78d254d2415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:35.087126 24206 consensus_queue.cc:1035] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd2c5f98eecd4b1d8271cfa46fd700f6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:35.087301 24208 consensus_queue.cc:1035] T ba78483b8dd9427aa6d71699208c073a P 8ca26d5b94ef47049610a78d254d2415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "010e38b9da5d4e05be07083caea7fdbd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:35.088016 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.087911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:57673)
0810 04:16:35.087952 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.087984 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 04:16:35.088903 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.088770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:40657)
0810 04:16:35.088818 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.088857 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:16:35.089656 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.089513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.65:36319)
0810 04:16:35.089581 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.089629 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 04:16:35.090014 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.089882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.66:51121)
0810 04:16:35.089953 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.089980 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 04:16:35.090457 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.090357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:34993 (local address 127.15.216.67:35693)
0810 04:16:35.090394 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.090423 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260810 04:16:35.092049 23624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:35.092142 23624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:35.093510 23624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7157b54a5b04db5b6f1d5b76356674b: Sending DeleteTablet for 3 replicas of tablet ba78483b8dd9427aa6d71699208c073a
I20260810 04:16:35.093875 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet ba78483b8dd9427aa6d71699208c073a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:35 UTC) from {username='oryx', principal='oryx/127.15.216.124@KRBTEST.COM'} at 127.0.0.1:47484
I20260810 04:16:35.093886 23885 tablet_service.cc:1514] Processing DeleteTablet for tablet ba78483b8dd9427aa6d71699208c073a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:35 UTC) from {username='oryx', principal='oryx/127.15.216.124@KRBTEST.COM'} at 127.0.0.1:44402
I20260810 04:16:35.094152 24222 tablet_replica.cc:330] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: stopping tablet replica
I20260810 04:16:35.094211 23742 tablet_service.cc:1514] Processing DeleteTablet for tablet ba78483b8dd9427aa6d71699208c073a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:35 UTC) from {username='oryx', principal='oryx/127.15.216.124@KRBTEST.COM'} at 127.0.0.1:59856
I20260810 04:16:35.094242 24222 raft_consensus.cc:2229] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:35.094295 24222 raft_consensus.cc:2258] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:35.094586 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 23583
I20260810 04:16:35.098459 24222 ts_tablet_manager.cc:1905] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:35.100304 24222 ts_tablet_manager.cc:1918] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:35.100383 24222 log.cc:1196] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/ba78483b8dd9427aa6d71699208c073a
I20260810 04:16:35.100652 24222 ts_tablet_manager.cc:1939] T ba78483b8dd9427aa6d71699208c073a P 010e38b9da5d4e05be07083caea7fdbd: Deleting consensus metadata
I20260810 04:16:35.101020 23608 catalog_manager.cc:4841] TS 010e38b9da5d4e05be07083caea7fdbd (127.15.216.67:37721): tablet ba78483b8dd9427aa6d71699208c073a (table test-table [id=ffcce051c9b048c4885a27a85dd2b889]) successfully deleted
I20260810 04:16:35.103314 24223 tablet_replica.cc:330] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6: stopping tablet replica
I20260810 04:16:35.103439 24223 raft_consensus.cc:2229] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:35.103506 24223 raft_consensus.cc:2258] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:35.103605 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.103488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:32781 (local address 127.0.0.1:59876)
0810 04:16:35.103536 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.103562 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 04:16:35.103806 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 23820
W20260810 04:16:35.103843 23609 catalog_manager.cc:4584] TS 8ca26d5b94ef47049610a78d254d2415 (127.15.216.65:32781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba78483b8dd9427aa6d71699208c073a: Network error: Client connection negotiation failed: client connection to 127.15.216.65:32781: connect: Connection refused (error 111)
I20260810 04:16:35.103863 24223 ts_tablet_manager.cc:1905] T ba78483b8dd9427aa6d71699208c073a P cd2c5f98eecd4b1d8271cfa46fd700f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:35.110916 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:35.110777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41677 (local address 127.0.0.1:44404)
0810 04:16:35.110840 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:35.110876 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.66:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 04:16:35.110936 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 23962
I20260810 04:16:35.119859 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 23288
I20260810 04:16:35.127483 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 23483
2026-08-10T04:16:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6626 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:35 dist-test-slave-g350 krb5kdc[24228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:35 dist-test-slave-g350 krb5kdc[24228](info): set up 2 sockets
Aug 10 04:16:35 dist-test-slave-g350 krb5kdc[24228](info): commencing operation
krb5kdc: starting...
W20260810 04:16:36.220350 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab.
Entry for principal oryx/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab.
I20260810 04:16:36.353253 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40801
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab
--principal=oryx/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:40801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:36.442022 24241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:36.442263 24241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:36.442298 24241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:36.443428 24241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:36.443495 24241 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 04:16:36.443517 24241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:36.443542 24241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:36.443571 24241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:36.443595 24241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:36.445176 24241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:40801
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.126.keytab
--principal=oryx/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:36.445487 24241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:36.445828 24241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:36.451583 24249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:36.451723 24241 server_base.cc:1034] running on GCE node
W20260810 04:16:36.451709 24247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:36.451583 24246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:36.452242 24241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:36.452643 24241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:36.453831 24241 hybrid_clock.cc:648] HybridClock initialized: now 1786335396453806 us; error 41 us; skew 500 ppm
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:36.457020 24241 init.cc:377] Logged in from keytab as oryx/127.15.216.126@KRBTEST.COM (short username oryx)
I20260810 04:16:36.457686 24241 webserver.cc:460] Webserver started at http://127.15.216.126:33223/ using document root <none> and password file <none>
I20260810 04:16:36.457952 24241 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:36.458007 24241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:36.458158 24241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:36.459444 24241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "cdb158e903ed44f9a08a72b85960c17c"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "c3c8d15697174577366f8db4a1caab4b"
server_key_iv: "260915aefb40b83600ba595b4654c817"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.459861 24241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "cdb158e903ed44f9a08a72b85960c17c"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "c3c8d15697174577366f8db4a1caab4b"
server_key_iv: "260915aefb40b83600ba595b4654c817"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.461480 24241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260810 04:16:36.462581 24257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:36.462855 24241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:36.462965 24241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "cdb158e903ed44f9a08a72b85960c17c"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "c3c8d15697174577366f8db4a1caab4b"
server_key_iv: "260915aefb40b83600ba595b4654c817"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.463032 24241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:36.478982 24241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:36.481683 24241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:36.481818 24241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:36.487123 24241 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:40801
I20260810 04:16:36.487339 24322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:40801 every 8 connection(s)
I20260810 04:16:36.487609 24241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:36.488416 24323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:36.489487 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 24241
I20260810 04:16:36.489624 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:36.489857 16225 external_mini_cluster.cc:1428] Setting key e9e2fb7cbd3d6f5d1c45a79e8be08161
I20260810 04:16:36.491333 24323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Bootstrap starting.
I20260810 04:16:36.492030 24323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:36.492385 24323 log.cc:824] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:36.493248 24323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: No bootstrap required, opened a new log
I20260810 04:16:36.495627 24323 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb158e903ed44f9a08a72b85960c17c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40801 } }
I20260810 04:16:36.495772 24323 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:36.495796 24323 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb158e903ed44f9a08a72b85960c17c, State: Initialized, Role: FOLLOWER
I20260810 04:16:36.495965 24323 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb158e903ed44f9a08a72b85960c17c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40801 } }
I20260810 04:16:36.496043 24323 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:36.496068 24323 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:36.496135 24323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:36.496666 24323 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb158e903ed44f9a08a72b85960c17c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40801 } }
I20260810 04:16:36.496763 24323 leader_election.cc:302] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb158e903ed44f9a08a72b85960c17c; no voters:
I20260810 04:16:36.496968 24323 leader_election.cc:288] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:36.497045 24328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:36.497201 24328 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [term 1 LEADER]: Becoming Leader. State: Replica: cdb158e903ed44f9a08a72b85960c17c, State: Running, Role: LEADER
I20260810 04:16:36.497243 24323 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:36.497301 24328 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb158e903ed44f9a08a72b85960c17c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40801 } }
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:36.497661 24328 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb158e903ed44f9a08a72b85960c17c. Latest consensus state: current_term: 1 leader_uuid: "cdb158e903ed44f9a08a72b85960c17c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb158e903ed44f9a08a72b85960c17c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40801 } } }
I20260810 04:16:36.497684 24329 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdb158e903ed44f9a08a72b85960c17c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb158e903ed44f9a08a72b85960c17c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40801 } } }
I20260810 04:16:36.497768 24329 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:36.497923 24328 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c [sys.catalog]: This master's current role is: LEADER
W20260810 04:16:36.499302 24345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:16:36.499363 24345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:16:36.499429 24335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:36.499621 24335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:36.501225 24335 catalog_manager.cc:1347] Generated new cluster ID: 86df69f0ef8b418b8b8e300f957f7351
I20260810 04:16:36.501292 24335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:36.502396 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.490492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.15.216.126:40801)
0810 04:16:36.490697 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:36.490707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:36.490744 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:16:36.491315 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:36.491325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:36.491388 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:36.491538 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:36.492291 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.493697 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.494446 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.495074 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.495192 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.498233 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:36.498238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:36.498238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:36.498250 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:36.500500 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.500990 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.500994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.500994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.501100 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.501394 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.501395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.501396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.501752 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:36.501940 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:36.502050 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 04:16:36.502203 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260810 04:16:36.513460 24335 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:36.514223 24335 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:36.528115 24335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Generated new TSK 0
I20260810 04:16:36.528352 24335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab.
Entry for principal oryx/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab.
I20260810 04:16:36.569231 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40801
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab
--principal=oryx/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:36.654261 24350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:36.654501 24350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:36.654531 24350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:36.655691 24350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:36.655736 24350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:36.655757 24350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:36.655777 24350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:36.657401 24350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab
--principal=oryx/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:36.657711 24350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:36.657948 24350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:36.663545 24355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:36.663579 24358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:36.663573 24356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:36.663705 24350 server_base.cc:1034] running on GCE node
I20260810 04:16:36.663949 24350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:36.664170 24350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:36.665418 24350 hybrid_clock.cc:648] HybridClock initialized: now 1786335396665389 us; error 31 us; skew 500 ppm
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:36.668346 24350 init.cc:377] Logged in from keytab as oryx/127.15.216.65@KRBTEST.COM (short username oryx)
I20260810 04:16:36.668838 24350 webserver.cc:460] Webserver started at http://127.15.216.65:38919/ using document root <none> and password file <none>
I20260810 04:16:36.669083 24350 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:36.669155 24350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:36.669322 24350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:36.670498 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "c544c740dfc441378335be3f453ff1bf"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "85d52f3ce7c7cf6dc3ea7a1d50a29321"
server_key_iv: "ae282523323aad9033c0ac4d730f42c6"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.670873 24350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "c544c740dfc441378335be3f453ff1bf"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "85d52f3ce7c7cf6dc3ea7a1d50a29321"
server_key_iv: "ae282523323aad9033c0ac4d730f42c6"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.672281 24350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:16:36.673120 24365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:36.673348 24350 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:36.673463 24350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "c544c740dfc441378335be3f453ff1bf"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "85d52f3ce7c7cf6dc3ea7a1d50a29321"
server_key_iv: "ae282523323aad9033c0ac4d730f42c6"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.673542 24350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:36.681133 24350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:36.683519 24350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:36.683713 24350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:36.684347 24350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:36.684419 24350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:36.684466 24350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:36.684499 24350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:36.692818 24350 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:44325
I20260810 04:16:36.692925 24484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:44325 every 8 connection(s)
I20260810 04:16:36.693233 24350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:36.695043 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 24350
I20260810 04:16:36.695169 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:36.695422 16225 external_mini_cluster.cc:1428] Setting key afff0516cdede547e9c050377a88b90b
I20260810 04:16:36.696085 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.684879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:59961 (local address 127.15.216.126:40801)
0810 04:16:36.684949 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:36.684952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:36.685180 (+ 228us) server_negotiation.cc:407] Connection header received
0810 04:16:36.685880 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:36.685882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:36.685897 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:36.685981 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:36.686983 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.687907 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.689335 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.689680 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.689783 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.692746 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:36.692748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:36.692749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:36.692757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:36.694414 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.694875 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.694876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.694876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.694929 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.695286 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.695287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.695287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.695419 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:36.695515 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:36.695879 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 04:16:36.695996 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:16:36.696355 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.684859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40801 (local address 127.15.216.65:59961)
0810 04:16:36.685040 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:36.685078 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:36.685760 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:36.686063 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:36.686068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:36.686488 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:36.686881 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.686889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.688008 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.688009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.689228 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.689231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.689819 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.689821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.689965 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.690545 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:36.690556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:36.692613 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:36.694497 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.694502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.694504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.694748 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.695002 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.695003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.695004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.695196 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.695508 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:36.695517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:36.695697 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 04:16:36.695845 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 04:16:36.696017 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260810 04:16:36.696887 24485 heartbeater.cc:347] Connected to a master server at 127.15.216.126:40801
I20260810 04:16:36.697007 24485 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:36.697234 24485 heartbeater.cc:511] Master 127.15.216.126:40801 requested a full tablet report, sending...
I20260810 04:16:36.697825 24275 ts_manager.cc:194] Registered new tserver with Master: c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325)
I20260810 04:16:36.698719 24275 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:59961
I20260810 04:16:36.704895 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.698514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:47933 (local address 127.15.216.126:40801)
0810 04:16:36.698569 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:36.698571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:36.698578 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:36.698596 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:36.698597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:36.698606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:36.698660 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:36.699124 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.699602 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.700659 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.700880 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.700975 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.702093 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:36.702094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:36.702095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:36.702102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:36.703632 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.704017 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.704018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.704019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.704081 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.704322 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.704323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.704323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.704447 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:36.704530 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:36.704709 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 04:16:36.704806 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:16:36.704907 24492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.698150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40801 (local address 127.15.216.65:47933)
0810 04:16:36.698305 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:36.698316 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:36.698367 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:36.698705 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:36.698707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:36.698956 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:36.699033 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.699035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.699723 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.699725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.700557 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.700560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.700979 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.700981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.701061 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.701622 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:36.701628 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:36.701975 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:36.703752 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.703753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.703753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.703941 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.704151 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.704152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.704152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.704217 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.704532 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:36.704533 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:36.704583 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 04:16:36.704666 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:16:36.704800 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":38,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab.
Entry for principal oryx/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab.
I20260810 04:16:36.725539 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40801
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab
--principal=oryx/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:36.815768 24494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:36.816020 24494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:36.816048 24494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:36.817281 24494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:36.817339 24494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:36.817363 24494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:36.817384 24494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:36.819172 24494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab
--principal=oryx/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:36.819471 24494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:36.819732 24494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:36.825697 24500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:36.825796 24499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:36.825706 24502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:36.826030 24494 server_base.cc:1034] running on GCE node
I20260810 04:16:36.826308 24494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:36.826658 24494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:36.827833 24494 hybrid_clock.cc:648] HybridClock initialized: now 1786335396827813 us; error 42 us; skew 500 ppm
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:36.830729 24494 init.cc:377] Logged in from keytab as oryx/127.15.216.66@KRBTEST.COM (short username oryx)
I20260810 04:16:36.831202 24494 webserver.cc:460] Webserver started at http://127.15.216.66:36879/ using document root <none> and password file <none>
I20260810 04:16:36.831487 24494 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:36.831560 24494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:36.831712 24494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:36.832780 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "2b0e263148fa4edc8103b1664c1d56af"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "ca13c70847b9bbad69941c0a44924ae8"
server_key_iv: "63191755904d671739efa1ba19818a4f"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.833137 24494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "2b0e263148fa4edc8103b1664c1d56af"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "ca13c70847b9bbad69941c0a44924ae8"
server_key_iv: "63191755904d671739efa1ba19818a4f"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.834600 24494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:36.835469 24509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:36.835669 24494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:36.835796 24494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "2b0e263148fa4edc8103b1664c1d56af"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "ca13c70847b9bbad69941c0a44924ae8"
server_key_iv: "63191755904d671739efa1ba19818a4f"
server_key_version: "encryptionkey@0"
I20260810 04:16:36.835875 24494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:36.849865 24494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:36.852288 24494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:36.852463 24494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:36.853102 24494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:36.853155 24494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:36.853190 24494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:36.853207 24494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:36.861070 24494 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:33007
I20260810 04:16:36.861202 24629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:33007 every 8 connection(s)
I20260810 04:16:36.861519 24494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:36.862040 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 24494
I20260810 04:16:36.862180 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:36.862408 16225 external_mini_cluster.cc:1428] Setting key e039ed226d93918743be36206eb860c2
I20260810 04:16:36.864003 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.853451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40801 (local address 127.15.216.66:42523)
0810 04:16:36.853653 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:36.853705 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:36.854236 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:36.854509 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:36.854513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:36.854830 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:36.855066 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.855072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.855780 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.855781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.856612 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.856615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.857068 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.857069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.857187 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.857732 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:36.857742 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:36.860381 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:36.862329 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.862341 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.862342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.862617 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.862898 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.862899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.862900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.863033 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.863490 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:36.863496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:36.863613 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 04:16:36.863701 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:16:36.863830 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260810 04:16:36.864001 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.853505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:42523 (local address 127.15.216.126:40801)
0810 04:16:36.853572 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:36.853574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:36.853790 (+ 216us) server_negotiation.cc:407] Connection header received
0810 04:16:36.854347 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:36.854349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:36.854362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:36.854436 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:36.855165 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.855674 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.856709 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.856963 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.857052 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.860489 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:36.860491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:36.860491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:36.860500 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:36.862219 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.862728 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.862730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.862730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.862808 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.863133 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.863135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.863141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.863398 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:36.863497 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:36.863754 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 04:16:36.863854 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:16:36.864493 24630 heartbeater.cc:347] Connected to a master server at 127.15.216.126:40801
I20260810 04:16:36.864586 24630 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:36.864785 24630 heartbeater.cc:511] Master 127.15.216.126:40801 requested a full tablet report, sending...
I20260810 04:16:36.865162 24275 ts_manager.cc:194] Registered new tserver with Master: 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007)
I20260810 04:16:36.865620 24275 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:42523
I20260810 04:16:36.873354 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.866031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:58821 (local address 127.15.216.126:40801)
0810 04:16:36.866067 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:36.866069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:36.866458 (+ 389us) server_negotiation.cc:407] Connection header received
0810 04:16:36.866482 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:36.866482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:36.866499 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:36.866548 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:36.867021 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.867595 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.868801 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.868978 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.869033 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.870210 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:36.870211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:36.870211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:36.870217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:36.871825 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.872421 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.872422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.872423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.872476 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:36.872747 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:36.872748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:36.872748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:36.872935 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:36.873000 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:36.873209 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 04:16:36.873280 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:16:36.873454 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:36.865877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40801 (local address 127.15.216.66:58821)
0810 04:16:36.866223 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:36.866237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:36.866332 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:36.866585 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:36.866586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:36.866852 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:36.866925 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.866928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.867707 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.867708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.868701 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:36.868705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:36.869063 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:36.869064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:36.869149 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:36.869639 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:36.869645 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:36.870118 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:36.871923 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.871925 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.871925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.872172 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.872561 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:36.872563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:36.872563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:36.872646 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:36.873022 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:36.873024 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:36.873082 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 04:16:36.873176 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 04:16:36.873298 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for oryx/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab.
Entry for principal oryx/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab.
I20260810 04:16:36.894506 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:40801
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab
--principal=oryx/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:36.985153 24640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:36.985416 24640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:36.985462 24640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:36.986689 24640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:36.986765 24640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:36.986814 24640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 04:16:36.986873 24640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:36.988487 24640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab
--principal=oryx/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.216.126:40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:36.988811 24640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:36.989063 24640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:16:36.994437 24640 server_base.cc:1034] running on GCE node
W20260810 04:16:36.994463 24648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:36.994582 24646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:36.994462 24645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:36.994849 24640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:36.995097 24640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:36.996275 24640 hybrid_clock.cc:648] HybridClock initialized: now 1786335396996284 us; error 59 us; skew 500 ppm
Aug 10 04:16:36 dist-test-slave-g350 krb5kdc[24228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:36.999265 24640 init.cc:377] Logged in from keytab as oryx/127.15.216.67@KRBTEST.COM (short username oryx)
I20260810 04:16:36.999755 24640 webserver.cc:460] Webserver started at http://127.15.216.67:36791/ using document root <none> and password file <none>
I20260810 04:16:37.000023 24640 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:37.000097 24640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:37.000275 24640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:37.001349 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "e282c2cd4a8f42378d67a53e805cf61a"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "a46d7d0fa07fc20e662b4da23cca34ac"
server_key_iv: "8afa13fc4ba5d6e1c40e58b44acccf26"
server_key_version: "encryptionkey@0"
I20260810 04:16:37.001753 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "e282c2cd4a8f42378d67a53e805cf61a"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "a46d7d0fa07fc20e662b4da23cca34ac"
server_key_iv: "8afa13fc4ba5d6e1c40e58b44acccf26"
server_key_version: "encryptionkey@0"
I20260810 04:16:37.003137 24640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:37.003929 24655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.004122 24640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:16:37.004227 24640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "e282c2cd4a8f42378d67a53e805cf61a"
format_stamp: "Formatted at 2026-08-10 04:16:36 on dist-test-slave-g350"
server_key: "a46d7d0fa07fc20e662b4da23cca34ac"
server_key_iv: "8afa13fc4ba5d6e1c40e58b44acccf26"
server_key_version: "encryptionkey@0"
I20260810 04:16:37.004315 24640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:37.019564 24640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:37.022197 24640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:37.022400 24640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:37.023072 24640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:37.023128 24640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.023165 24640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:37.023190 24640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:37.031234 24640 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:34597
I20260810 04:16:37.031353 24779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:34597 every 8 connection(s)
I20260810 04:16:37.031728 24640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:37.034260 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.023556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:38801 (local address 127.15.216.126:40801)
0810 04:16:37.023617 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.023619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.023909 (+ 290us) server_negotiation.cc:407] Connection header received
0810 04:16:37.024670 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.024671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.024683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.024729 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:37.025714 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.026241 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.027322 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.027535 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.027614 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.031302 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:37.031304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:37.031305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:37.031313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:37.032848 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.033235 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.033236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.033236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.033336 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.033643 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.033644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.033645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.033774 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:37.033829 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.034117 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.034197 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:16:37.034320 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.023531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40801 (local address 127.15.216.67:38801)
0810 04:16:37.023750 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.023804 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.024557 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.024816 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.024822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.025238 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:37.025572 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.025579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.026354 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.026355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.027221 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.027224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.027626 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.027627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.027754 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.028518 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:37.028547 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:37.031166 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:37.032925 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:37.032931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:37.032933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:37.033161 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:37.033422 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:37.033423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:37.033423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:37.033526 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:37.033832 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:37.033837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:37.033954 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.034040 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.034162 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260810 04:16:37.034840 24780 heartbeater.cc:347] Connected to a master server at 127.15.216.126:40801
I20260810 04:16:37.034950 24780 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:37.035137 24780 heartbeater.cc:511] Master 127.15.216.126:40801 requested a full tablet report, sending...
I20260810 04:16:37.035542 24276 ts_manager.cc:194] Registered new tserver with Master: e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597)
I20260810 04:16:37.035941 24276 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:38801
I20260810 04:16:37.041208 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 24640
I20260810 04:16:37.041241 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.035984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40801 (local address 127.15.216.67:54811)
0810 04:16:37.036102 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.036114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.036168 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.036305 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.036306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.036515 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:37.036584 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.036586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.037204 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.037205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.037784 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.037786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.038088 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.038089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.038133 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.038621 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:37.038627 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:37.038952 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:37.040405 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:37.040406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:37.040406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:37.040566 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:37.040737 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:37.040737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:37.040737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:37.040788 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:37.041002 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:37.041002 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:37.041039 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.041090 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.041170 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260810 04:16:37.041321 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.036111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:54811 (local address 127.15.216.126:40801)
0810 04:16:37.036152 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.036155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.036163 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:37.036216 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.036217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.036227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.036284 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:37.036661 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.037117 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.037853 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.038024 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.038076 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.039026 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:37.039027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:37.039027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:37.039033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:37.040330 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.040632 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.040633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.040633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.040681 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.040842 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.040842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.040843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.040933 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:37.041003 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.041143 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.041219 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:16:37.041329 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:37.041535 16225 external_mini_cluster.cc:1428] Setting key 8e4757258a55e8244c01678816e01e86
I20260810 04:16:37.042474 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:16:37.066269 24326 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:37.060133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.15.216.126:40801)
0810 04:16:37.060198 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.060200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.060212 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:16:37.060353 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.060354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.060362 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.060415 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:37.060993 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.061481 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.062345 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.062533 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.062599 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.066133 (+ 3534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38212: BlockingRecv error: recv got EOF from 127.0.0.1:38212 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:37.074460 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.067157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.15.216.126:40801)
0810 04:16:37.067222 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.067224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.067322 (+ 98us) server_negotiation.cc:407] Connection header received
0810 04:16:37.067376 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.067377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.067393 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.067473 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:37.067990 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.068469 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.069166 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.069362 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.069439 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.071697 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:37.071704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:37.071704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:37.071711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:37.073157 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.073563 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.073565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.073565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.073708 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.073935 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.073936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.073937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.074055 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:37.074166 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.074268 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.074351 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:16:37.076625 24276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:37.078501 24276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:37.085899 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.081212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:44325 (local address 127.0.0.1:55456)
0810 04:16:37.081714 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.081726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.081779 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.082313 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.082314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.082316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.082353 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.082355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.083591 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.083592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.084596 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.084599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.085537 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.085538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.085653 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.085671 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.085748 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.085806 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260810 04:16:37.086059 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.081519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.15.216.66:33007)
0810 04:16:37.081770 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.081775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.081786 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:37.081830 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.081832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.081849 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.082154 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.082233 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.083377 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.084885 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.085649 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.085744 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.085818 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.085924 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.085964 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":147,"threads_started":1}
I20260810 04:16:37.086437 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.081515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.15.216.65:44325)
0810 04:16:37.082098 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.082102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.082112 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:37.082151 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.082153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.082168 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.082241 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.082429 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.083504 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.084700 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.085445 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.085526 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.085636 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.086284 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.086328 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":457,"threads_started":1}
I20260810 04:16:37.086038 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.081209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:33007 (local address 127.0.0.1:34364)
0810 04:16:37.081363 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.081407 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.081494 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.081972 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.081975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.081978 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.082065 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.082068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.083455 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.083455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.084794 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.084795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.085759 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.085760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.085836 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.085841 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.085907 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.085952 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":31,"thread_start_us":73,"threads_started":1}
I20260810 04:16:37.086902 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.081212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34597 (local address 127.0.0.1:43202)
0810 04:16:37.081863 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.081870 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.081930 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.082373 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.082374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.082376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.082466 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.082468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.083391 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.083392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.084791 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.084795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.086667 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.086668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.086740 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.086745 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.086789 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.086829 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":51,"thread_start_us":100,"threads_started":1}
I20260810 04:16:37.087260 24560 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2206a187eb47f08f593a5ce48f35f2 (DEFAULT_TABLE table=test-table [id=9de7812245fa4f84acd9c3b7e71c495a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.087296 24415 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2206a187eb47f08f593a5ce48f35f2 (DEFAULT_TABLE table=test-table [id=9de7812245fa4f84acd9c3b7e71c495a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.087579 24560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2206a187eb47f08f593a5ce48f35f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.087656 24415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2206a187eb47f08f593a5ce48f35f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.091588 24809 tablet_bootstrap.cc:492] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Bootstrap starting.
I20260810 04:16:37.091735 24810 tablet_bootstrap.cc:492] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Bootstrap starting.
I20260810 04:16:37.091933 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.081781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.15.216.67:34597)
0810 04:16:37.082170 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.082175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.082185 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:37.082269 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.082272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.082286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.082376 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.082570 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.083294 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.085695 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.086502 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.086581 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.086766 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.086803 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.086844 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":103,"threads_started":1}
I20260810 04:16:37.092424 24809 tablet_bootstrap.cc:654] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.092422 24810 tablet_bootstrap.cc:654] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.092783 24810 log.cc:824] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:37.092643 24710 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2206a187eb47f08f593a5ce48f35f2 (DEFAULT_TABLE table=test-table [id=9de7812245fa4f84acd9c3b7e71c495a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.092797 24809 log.cc:824] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:37.093047 24710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2206a187eb47f08f593a5ce48f35f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.093657 24809 tablet_bootstrap.cc:492] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: No bootstrap required, opened a new log
I20260810 04:16:37.093673 24810 tablet_bootstrap.cc:492] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: No bootstrap required, opened a new log
I20260810 04:16:37.093747 24809 ts_tablet_manager.cc:1397] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:16:37.093747 24810 ts_tablet_manager.cc:1397] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:37.095860 24809 raft_consensus.cc:348] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.096061 24809 raft_consensus.cc:374] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.096107 24809 raft_consensus.cc:729] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c544c740dfc441378335be3f453ff1bf, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.096000 24813 tablet_bootstrap.cc:492] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Bootstrap starting.
I20260810 04:16:37.096256 24809 consensus_queue.cc:260] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.096253 24810 raft_consensus.cc:348] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.096392 24810 raft_consensus.cc:374] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.096446 24810 raft_consensus.cc:729] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0e263148fa4edc8103b1664c1d56af, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.096553 24809 ts_tablet_manager.cc:1428] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:16:37.096594 24810 consensus_queue.cc:260] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.096726 24485 heartbeater.cc:503] Master 127.15.216.126:40801 was elected leader, sending a full tablet report...
I20260810 04:16:37.096869 24810 ts_tablet_manager.cc:1428] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:16:37.096895 24813 tablet_bootstrap.cc:654] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.096982 24630 heartbeater.cc:503] Master 127.15.216.126:40801 was elected leader, sending a full tablet report...
I20260810 04:16:37.097318 24813 log.cc:824] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:37.098193 24813 tablet_bootstrap.cc:492] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: No bootstrap required, opened a new log
I20260810 04:16:37.098294 24813 ts_tablet_manager.cc:1397] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:37.100045 24813 raft_consensus.cc:348] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.100172 24813 raft_consensus.cc:374] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.100194 24813 raft_consensus.cc:729] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282c2cd4a8f42378d67a53e805cf61a, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.100307 24813 consensus_queue.cc:260] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.100584 24813 ts_tablet_manager.cc:1428] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:16:37.100646 24780 heartbeater.cc:503] Master 127.15.216.126:40801 was elected leader, sending a full tablet report...
W20260810 04:16:37.112332 24631 tablet.cc:2369] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:37.172688 24814 raft_consensus.cc:482] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:37.172897 24814 raft_consensus.cc:504] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.173334 24814 leader_election.cc:288] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007), e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597)
I20260810 04:16:37.176976 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.173678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34597 (local address 127.15.216.65:53611)
0810 04:16:37.173761 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.173777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.173834 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.174186 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.174188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.174190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.174282 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.174283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.175125 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.175127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.176020 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.176023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.176632 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.176634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.176748 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.176752 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.176806 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.176884 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":2}
I20260810 04:16:37.177309 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.173954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:53611 (local address 127.15.216.67:34597)
0810 04:16:37.174022 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.174024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.174034 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:37.174063 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.174064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.174073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.174160 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.174528 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.175018 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.176118 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.176557 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.176627 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.176637 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.177195 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.177246 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:16:37.177357 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.173919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:57877 (local address 127.15.216.66:33007)
0810 04:16:37.173985 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.173988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.173998 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:37.174032 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.174033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.174049 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.174119 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.174362 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.174896 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.176134 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.176808 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.176919 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.176929 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.177236 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.177284 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:16:37.177404 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.173714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:33007 (local address 127.15.216.65:57877)
0810 04:16:37.173888 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.173898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.173944 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.174142 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.174143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.174145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.174242 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.174245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.175016 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.175018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.176018 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.176021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.177030 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.177031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.177110 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.177115 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.177218 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.177264 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260810 04:16:37.177670 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2206a187eb47f08f593a5ce48f35f2" candidate_uuid: "c544c740dfc441378335be3f453ff1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282c2cd4a8f42378d67a53e805cf61a" is_pre_election: true
I20260810 04:16:37.177856 24732 raft_consensus.cc:2454] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c544c740dfc441378335be3f453ff1bf in term 0.
I20260810 04:16:37.177902 24581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2206a187eb47f08f593a5ce48f35f2" candidate_uuid: "c544c740dfc441378335be3f453ff1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0e263148fa4edc8103b1664c1d56af" is_pre_election: true
I20260810 04:16:37.178138 24366 leader_election.cc:302] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c544c740dfc441378335be3f453ff1bf, e282c2cd4a8f42378d67a53e805cf61a; no voters:
I20260810 04:16:37.178081 24581 raft_consensus.cc:2454] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c544c740dfc441378335be3f453ff1bf in term 0.
I20260810 04:16:37.178318 24814 raft_consensus.cc:2781] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:37.178468 24814 raft_consensus.cc:482] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:37.178535 24814 raft_consensus.cc:3037] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.179597 24814 raft_consensus.cc:504] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.179787 24814 leader_election.cc:288] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [CANDIDATE]: Term 1 election: Requested vote from peers 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007), e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597)
I20260810 04:16:37.179965 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2206a187eb47f08f593a5ce48f35f2" candidate_uuid: "c544c740dfc441378335be3f453ff1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282c2cd4a8f42378d67a53e805cf61a"
I20260810 04:16:37.180070 24732 raft_consensus.cc:3037] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.180138 24581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2206a187eb47f08f593a5ce48f35f2" candidate_uuid: "c544c740dfc441378335be3f453ff1bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0e263148fa4edc8103b1664c1d56af"
I20260810 04:16:37.180238 24581 raft_consensus.cc:3037] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.181000 24581 raft_consensus.cc:2454] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c544c740dfc441378335be3f453ff1bf in term 1.
I20260810 04:16:37.181089 24732 raft_consensus.cc:2454] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c544c740dfc441378335be3f453ff1bf in term 1.
I20260810 04:16:37.181207 24369 leader_election.cc:302] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, c544c740dfc441378335be3f453ff1bf; no voters:
I20260810 04:16:37.181315 24814 raft_consensus.cc:2781] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:37.181658 24814 raft_consensus.cc:686] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 1 LEADER]: Becoming Leader. State: Replica: c544c740dfc441378335be3f453ff1bf, State: Running, Role: LEADER
I20260810 04:16:37.181816 24814 consensus_queue.cc:237] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } }
I20260810 04:16:37.183064 24276 catalog_manager.cc:5495] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf reported cstate change: term changed from 0 to 1, leader changed from <none> to c544c740dfc441378335be3f453ff1bf (127.15.216.65). New cstate: current_term: 1 leader_uuid: "c544c740dfc441378335be3f453ff1bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } health_report { overall_health: HEALTHY } } }
W20260810 04:16:37.194650 24486 tablet.cc:2369] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:37.206121 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.203188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.15.216.65:44325)
0810 04:16:37.203297 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.203299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.203308 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:37.203348 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.203349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.203368 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.203433 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:37.203599 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.204077 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.204881 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.205072 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.205129 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.205485 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.205670 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.205760 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.205908 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.205990 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 04:16:37.208479 24581 raft_consensus.cc:1261] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Refusing update from remote peer c544c740dfc441378335be3f453ff1bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.208479 24732 raft_consensus.cc:1261] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Refusing update from remote peer c544c740dfc441378335be3f453ff1bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.208871 24814 consensus_queue.cc:1035] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.209097 24819 consensus_queue.cc:1035] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.211876 24822 mvcc.cc:205] Tried to move back new op lower bound from 7316829786962542592 to 7316829786858016768. Current Snapshot: MvccSnapshot[applied={T|T < 7316829786962542592}]
I20260810 04:16:37.213747 24824 mvcc.cc:205] Tried to move back new op lower bound from 7316829786962542592 to 7316829786858016768. Current Snapshot: MvccSnapshot[applied={T|T < 7316829786962542592 or (T in {7316829786962542592})}]
I20260810 04:16:37.214589 24276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.214704 24276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.215574 24276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Sending DeleteTablet for 3 replicas of tablet 8f2206a187eb47f08f593a5ce48f35f2
I20260810 04:16:37.215987 24560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2206a187eb47f08f593a5ce48f35f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:34364
I20260810 04:16:37.216078 24710 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2206a187eb47f08f593a5ce48f35f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43202
I20260810 04:16:37.216316 24415 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f2206a187eb47f08f593a5ce48f35f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:55456
I20260810 04:16:37.216480 24831 tablet_replica.cc:330] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: stopping tablet replica
I20260810 04:16:37.216538 24833 tablet_replica.cc:330] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: stopping tablet replica
I20260810 04:16:37.216475 24276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 04:16:37.216640 24833 raft_consensus.cc:2229] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 1 LEADER]: Raft consensus shutting down.
W20260810 04:16:37.216660 24276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:37.216740 24833 raft_consensus.cc:2258] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.216926 24832 tablet_replica.cc:330] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: stopping tablet replica
I20260810 04:16:37.217037 24832 raft_consensus.cc:2229] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.217178 24832 raft_consensus.cc:2258] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.217702 24833 ts_tablet_manager.cc:1905] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.217705 24831 raft_consensus.cc:2229] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.218214 24831 raft_consensus.cc:2258] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.218339 24560 tablet_service.cc:1467] Processing CreateTablet for tablet 30fc3de2312f4878863da6eb4304f7d1 (DEFAULT_TABLE table=test-table [id=17177e82087e49c9b37b832bf3438ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.218485 24560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fc3de2312f4878863da6eb4304f7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.218935 24831 ts_tablet_manager.cc:1905] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.219965 24832 ts_tablet_manager.cc:1905] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.220088 24810 tablet_bootstrap.cc:492] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: Bootstrap starting.
I20260810 04:16:37.220410 24415 tablet_service.cc:1467] Processing CreateTablet for tablet 30fc3de2312f4878863da6eb4304f7d1 (DEFAULT_TABLE table=test-table [id=17177e82087e49c9b37b832bf3438ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.220635 24810 tablet_bootstrap.cc:654] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.220811 24710 tablet_service.cc:1467] Processing CreateTablet for tablet 30fc3de2312f4878863da6eb4304f7d1 (DEFAULT_TABLE table=test-table [id=17177e82087e49c9b37b832bf3438ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.220947 24710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fc3de2312f4878863da6eb4304f7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.222280 24831 ts_tablet_manager.cc:1918] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:37.222437 24831 log.cc:1196] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/8f2206a187eb47f08f593a5ce48f35f2
I20260810 04:16:37.222509 24810 tablet_bootstrap.cc:492] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: No bootstrap required, opened a new log
I20260810 04:16:37.222585 24810 ts_tablet_manager.cc:1397] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 04:16:37.222720 24813 tablet_bootstrap.cc:492] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Bootstrap starting.
I20260810 04:16:37.222769 24810 raft_consensus.cc:348] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.222849 24810 raft_consensus.cc:374] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.222872 24810 raft_consensus.cc:729] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0e263148fa4edc8103b1664c1d56af, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.222925 24810 consensus_queue.cc:260] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.223021 24810 ts_tablet_manager.cc:1428] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.223212 24813 tablet_bootstrap.cc:654] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.223414 24832 ts_tablet_manager.cc:1918] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:37.223481 24832 log.cc:1196] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/8f2206a187eb47f08f593a5ce48f35f2
I20260810 04:16:37.223590 24831 ts_tablet_manager.cc:1939] T 8f2206a187eb47f08f593a5ce48f35f2 P 2b0e263148fa4edc8103b1664c1d56af: Deleting consensus metadata
I20260810 04:16:37.223766 24832 ts_tablet_manager.cc:1939] T 8f2206a187eb47f08f593a5ce48f35f2 P e282c2cd4a8f42378d67a53e805cf61a: Deleting consensus metadata
I20260810 04:16:37.223968 24415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fc3de2312f4878863da6eb4304f7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.224292 24263 catalog_manager.cc:4841] TS 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007): tablet 8f2206a187eb47f08f593a5ce48f35f2 (table test-table [id=9de7812245fa4f84acd9c3b7e71c495a]) successfully deleted
I20260810 04:16:37.225385 24809 tablet_bootstrap.cc:492] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: Bootstrap starting.
I20260810 04:16:37.225481 24833 ts_tablet_manager.cc:1918] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:37.225560 24833 log.cc:1196] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/8f2206a187eb47f08f593a5ce48f35f2
I20260810 04:16:37.225883 24833 ts_tablet_manager.cc:1939] T 8f2206a187eb47f08f593a5ce48f35f2 P c544c740dfc441378335be3f453ff1bf: Deleting consensus metadata
I20260810 04:16:37.225992 24809 tablet_bootstrap.cc:654] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.226307 24260 catalog_manager.cc:4841] TS c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325): tablet 8f2206a187eb47f08f593a5ce48f35f2 (table test-table [id=9de7812245fa4f84acd9c3b7e71c495a]) successfully deleted
I20260810 04:16:37.226631 24259 catalog_manager.cc:4841] TS e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597): tablet 8f2206a187eb47f08f593a5ce48f35f2 (table test-table [id=9de7812245fa4f84acd9c3b7e71c495a]) successfully deleted
I20260810 04:16:37.226931 24809 tablet_bootstrap.cc:492] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: No bootstrap required, opened a new log
I20260810 04:16:37.227005 24809 ts_tablet_manager.cc:1397] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:37.227180 24809 raft_consensus.cc:348] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.227267 24809 raft_consensus.cc:374] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.227304 24809 raft_consensus.cc:729] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c544c740dfc441378335be3f453ff1bf, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.227360 24809 consensus_queue.cc:260] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.227470 24809 ts_tablet_manager.cc:1428] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 04:16:37.230103 24813 tablet_bootstrap.cc:492] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: No bootstrap required, opened a new log
I20260810 04:16:37.230161 24813 ts_tablet_manager.cc:1397] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 04:16:37.230353 24813 raft_consensus.cc:348] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.230461 24813 raft_consensus.cc:374] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.230479 24813 raft_consensus.cc:729] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282c2cd4a8f42378d67a53e805cf61a, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.230548 24813 consensus_queue.cc:260] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.230681 24813 ts_tablet_manager.cc:1428] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260810 04:16:37.282387 24781 tablet.cc:2369] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:16:37.300002 24817 raft_consensus.cc:482] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:37.300158 24817 raft_consensus.cc:504] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.300549 24817 leader_election.cc:288] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007)
I20260810 04:16:37.304236 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.300921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:33531 (local address 127.15.216.66:33007)
0810 04:16:37.300970 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.300972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.301020 (+ 48us) server_negotiation.cc:407] Connection header received
0810 04:16:37.301137 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.301138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.301148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.301215 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.301507 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.302182 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.303194 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.303747 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.303825 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.303834 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.304096 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.304147 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:16:37.304242 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.300737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:33007 (local address 127.15.216.67:33531)
0810 04:16:37.300960 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.300970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.301052 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.301272 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.301274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.301276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.301388 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.301390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.302276 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.302278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.303090 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.303093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.303877 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.303878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.303951 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.303957 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.304060 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.304105 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":66,"threads_started":1}
I20260810 04:16:37.304409 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.300829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:35903 (local address 127.15.216.65:44325)
0810 04:16:37.300876 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.300878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.300887 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:37.300917 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.300917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.300929 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.300986 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.301183 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.301862 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.303282 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.303869 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.303961 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.303976 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.304276 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.304325 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:16:37.304246 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.300716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:44325 (local address 127.15.216.67:35903)
0810 04:16:37.300775 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.300788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.300837 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.301040 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.301042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.301043 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.301121 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.301123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.301969 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.301971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.303182 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.303186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.303971 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.303972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.304069 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.304073 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.304129 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.304168 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:16:37.304651 24581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc3de2312f4878863da6eb4304f7d1" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0e263148fa4edc8103b1664c1d56af" is_pre_election: true
I20260810 04:16:37.304757 24581 raft_consensus.cc:2454] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 0.
I20260810 04:16:37.305022 24660 leader_election.cc:302] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, e282c2cd4a8f42378d67a53e805cf61a; no voters:
I20260810 04:16:37.305226 24817 raft_consensus.cc:2781] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:37.305284 24817 raft_consensus.cc:482] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:37.305308 24817 raft_consensus.cc:3037] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.305406 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc3de2312f4878863da6eb4304f7d1" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf" is_pre_election: true
I20260810 04:16:37.305562 24436 raft_consensus.cc:2454] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 0.
I20260810 04:16:37.306245 24817 raft_consensus.cc:504] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.306417 24817 leader_election.cc:288] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 election: Requested vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007)
I20260810 04:16:37.306609 24581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc3de2312f4878863da6eb4304f7d1" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0e263148fa4edc8103b1664c1d56af"
I20260810 04:16:37.306699 24581 raft_consensus.cc:3037] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.307338 24581 raft_consensus.cc:2454] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 1.
I20260810 04:16:37.307528 24660 leader_election.cc:302] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, e282c2cd4a8f42378d67a53e805cf61a; no voters:
I20260810 04:16:37.307613 24817 raft_consensus.cc:2781] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:37.307574 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc3de2312f4878863da6eb4304f7d1" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf"
I20260810 04:16:37.307654 24436 raft_consensus.cc:3037] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.307775 24817 raft_consensus.cc:686] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 1 LEADER]: Becoming Leader. State: Replica: e282c2cd4a8f42378d67a53e805cf61a, State: Running, Role: LEADER
I20260810 04:16:37.307858 24817 consensus_queue.cc:237] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.308549 24436 raft_consensus.cc:2454] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 1.
I20260810 04:16:37.309796 24276 catalog_manager.cc:5495] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a reported cstate change: term changed from 0 to 1, leader changed from <none> to e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67). New cstate: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } health_report { overall_health: HEALTHY } } }
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335396, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.126@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:37.347146 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.338515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:40801 (local address 127.0.0.1:38232)
0810 04:16:37.338704 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.338715 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.338773 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.338959 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.338960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.339387 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:37.339448 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.339451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.340413 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.340414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.341027 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.341029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.341507 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.341508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.341561 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.342311 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:37.342318 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:37.344055 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:37.345829 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:37.345838 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:37.345840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:37.346063 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:37.346321 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:37.346322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:37.346323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:37.346395 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:37.346748 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:37.346757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:37.346817 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.346916 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.347039 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260810 04:16:37.347203 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.338708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.15.216.126:40801)
0810 04:16:37.338779 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.338781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.338824 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:16:37.338850 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.338851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.338861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.338932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:37.339545 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.340337 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.341126 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.341411 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.341488 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.344160 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:37.344162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:37.344163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:37.344172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:37.345698 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.346146 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.346146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.346147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.346205 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.346497 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.346498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.346498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.346630 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:37.346696 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.346963 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.347090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":45}
I20260810 04:16:37.349658 24276 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:38232:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 04:16:37.349871 24276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:37.351493 24415 tablet_service.cc:1467] Processing CreateTablet for tablet 6441b1b652bc4f869e7554073a7b4ffc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87b0273415b44b4c883ad1c082d7da79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:37.351595 24560 tablet_service.cc:1467] Processing CreateTablet for tablet 6441b1b652bc4f869e7554073a7b4ffc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87b0273415b44b4c883ad1c082d7da79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:37.351680 24415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6441b1b652bc4f869e7554073a7b4ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.351665 24710 tablet_service.cc:1467] Processing CreateTablet for tablet 6441b1b652bc4f869e7554073a7b4ffc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87b0273415b44b4c883ad1c082d7da79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 04:16:37.351759 24560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6441b1b652bc4f869e7554073a7b4ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.351806 24710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6441b1b652bc4f869e7554073a7b4ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.353075 24810 tablet_bootstrap.cc:492] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: Bootstrap starting.
I20260810 04:16:37.353361 24813 tablet_bootstrap.cc:492] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: Bootstrap starting.
I20260810 04:16:37.353363 24809 tablet_bootstrap.cc:492] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: Bootstrap starting.
I20260810 04:16:37.353574 24810 tablet_bootstrap.cc:654] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.353710 24809 tablet_bootstrap.cc:654] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.353853 24813 tablet_bootstrap.cc:654] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.354503 24810 tablet_bootstrap.cc:492] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: No bootstrap required, opened a new log
I20260810 04:16:37.354559 24810 ts_tablet_manager.cc:1397] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:16:37.354688 24809 tablet_bootstrap.cc:492] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: No bootstrap required, opened a new log
I20260810 04:16:37.354725 24809 ts_tablet_manager.cc:1397] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:37.354709 24810 raft_consensus.cc:348] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.354763 24810 raft_consensus.cc:374] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.354776 24810 raft_consensus.cc:729] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0e263148fa4edc8103b1664c1d56af, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.354825 24809 raft_consensus.cc:348] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.354838 24810 consensus_queue.cc:260] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.354874 24809 raft_consensus.cc:374] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.354888 24809 raft_consensus.cc:729] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c544c740dfc441378335be3f453ff1bf, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.354930 24810 ts_tablet_manager.cc:1428] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.354957 24809 consensus_queue.cc:260] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.355054 24809 ts_tablet_manager.cc:1428] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.355054 24815 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.355139 24814 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.355158 24815 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:37.355203 24814 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:37.355495 24813 tablet_bootstrap.cc:492] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: No bootstrap required, opened a new log
I20260810 04:16:37.355556 24813 ts_tablet_manager.cc:1397] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:37.355726 24813 raft_consensus.cc:348] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.355813 24813 raft_consensus.cc:374] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.355839 24813 raft_consensus.cc:729] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282c2cd4a8f42378d67a53e805cf61a, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.355893 24813 consensus_queue.cc:260] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.356006 24813 ts_tablet_manager.cc:1428] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.356279 24840 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.356367 24840 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:37.578714 24840 raft_consensus.cc:482] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:37.578845 24840 raft_consensus.cc:504] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.579063 24840 leader_election.cc:288] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007)
I20260810 04:16:37.579420 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441b1b652bc4f869e7554073a7b4ffc" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf" is_pre_election: true
I20260810 04:16:37.579455 24581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441b1b652bc4f869e7554073a7b4ffc" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0e263148fa4edc8103b1664c1d56af" is_pre_election: true
I20260810 04:16:37.579547 24581 raft_consensus.cc:2454] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 0.
I20260810 04:16:37.579545 24436 raft_consensus.cc:2454] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 0.
I20260810 04:16:37.579747 24657 leader_election.cc:302] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c544c740dfc441378335be3f453ff1bf, e282c2cd4a8f42378d67a53e805cf61a; no voters:
I20260810 04:16:37.579901 24840 raft_consensus.cc:2781] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:37.579962 24840 raft_consensus.cc:482] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:37.579998 24840 raft_consensus.cc:3037] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.580860 24840 raft_consensus.cc:504] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.581023 24840 leader_election.cc:288] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 election: Requested vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007)
I20260810 04:16:37.581203 24581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441b1b652bc4f869e7554073a7b4ffc" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0e263148fa4edc8103b1664c1d56af"
I20260810 04:16:37.581205 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441b1b652bc4f869e7554073a7b4ffc" candidate_uuid: "e282c2cd4a8f42378d67a53e805cf61a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf"
I20260810 04:16:37.581300 24436 raft_consensus.cc:3037] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.581300 24581 raft_consensus.cc:3037] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.581959 24581 raft_consensus.cc:2454] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 1.
I20260810 04:16:37.582109 24436 raft_consensus.cc:2454] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e282c2cd4a8f42378d67a53e805cf61a in term 1.
I20260810 04:16:37.582261 24660 leader_election.cc:302] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, e282c2cd4a8f42378d67a53e805cf61a; no voters:
I20260810 04:16:37.582379 24840 raft_consensus.cc:2781] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:37.582482 24840 raft_consensus.cc:686] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [term 1 LEADER]: Becoming Leader. State: Replica: e282c2cd4a8f42378d67a53e805cf61a, State: Running, Role: LEADER
I20260810 04:16:37.582545 24840 consensus_queue.cc:237] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.582854 24817 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: TxnStatusTablet state changed. Reason: New leader e282c2cd4a8f42378d67a53e805cf61a. Latest consensus state: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.582947 24817 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:37.583139 24853 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 04:16:37.583189 24276 catalog_manager.cc:5495] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a reported cstate change: term changed from 0 to 1, leader changed from <none> to e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67). New cstate: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } health_report { overall_health: HEALTHY } } }
I20260810 04:16:37.583302 24853 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 04:16:37.642729 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.639773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34597 (local address 127.0.0.1:43204)
0810 04:16:37.639903 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.639916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.639980 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.640354 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.640355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.640357 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:37.640415 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.640417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.641102 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.641103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.641768 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.641770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.642145 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.642146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.642220 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.642239 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.642530 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.642541 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.642592 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.642644 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 04:16:37.642705 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.639847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.15.216.67:34597)
0810 04:16:37.639906 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.639908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.639954 (+ 46us) server_negotiation.cc:407] Connection header received
0810 04:16:37.640056 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.640058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.640140 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.640338 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:37.640504 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.641010 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.641856 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.642070 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.642146 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.642348 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.642457 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.642545 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.642597 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.642620 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:16:37.643989 24581 raft_consensus.cc:1261] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Refusing update from remote peer e282c2cd4a8f42378d67a53e805cf61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.644110 24436 raft_consensus.cc:1261] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Refusing update from remote peer e282c2cd4a8f42378d67a53e805cf61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.644315 24840 consensus_queue.cc:1035] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.644423 24817 consensus_queue.cc:1035] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.645051 24815 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: TxnStatusTablet state changed. Reason: New leader e282c2cd4a8f42378d67a53e805cf61a. Latest consensus state: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.645151 24815 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:37.645114 24814 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: TxnStatusTablet state changed. Reason: New leader e282c2cd4a8f42378d67a53e805cf61a. Latest consensus state: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.645203 24814 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:37.646020 24817 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.646109 24817 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:37.646210 24817 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.646263 24817 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P e282c2cd4a8f42378d67a53e805cf61a: This TxnStatusTablet replica's current role is: LEADER
I20260810 04:16:37.646613 24815 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.646700 24815 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P 2b0e263148fa4edc8103b1664c1d56af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:37.646924 24814 tablet_replica.cc:439] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e282c2cd4a8f42378d67a53e805cf61a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } } }
I20260810 04:16:37.647022 24814 tablet_replica.cc:442] T 6441b1b652bc4f869e7554073a7b4ffc P c544c740dfc441378335be3f453ff1bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 04:16:37.650718 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.648353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.15.216.67:34597)
0810 04:16:37.648392 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.648395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.648438 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:16:37.648515 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.648516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.648527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.648571 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:37.648697 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.649186 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.650005 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.650230 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.650310 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.650460 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.650496 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.650546 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.650610 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.650632 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:16:37.656968 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.654399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:37559 (local address 127.15.216.67:34597)
0810 04:16:37.654450 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.654452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.654549 (+ 97us) server_negotiation.cc:407] Connection header received
0810 04:16:37.654616 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.654618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.654629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.654675 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:37.654863 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.655341 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.656173 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.656412 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.656493 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.656681 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.656723 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.656792 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.656844 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.656869 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":27}
I20260810 04:16:37.657087 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.654293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34597 (local address 127.15.216.67:37559)
0810 04:16:37.654461 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.654474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.654567 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.654686 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.654687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.654688 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:37.654778 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.654780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.655429 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.655430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.656073 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.656075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.656494 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.656495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.656556 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.656581 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.656794 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.656796 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.656844 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.656890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260810 04:16:37.660271 24436 raft_consensus.cc:1261] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Refusing update from remote peer e282c2cd4a8f42378d67a53e805cf61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.660271 24581 raft_consensus.cc:1261] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Refusing update from remote peer e282c2cd4a8f42378d67a53e805cf61a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.660522 24861 consensus_queue.cc:1035] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.660706 24817 consensus_queue.cc:1035] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.675707 24275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.675874 24275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.676723 24275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Sending DeleteTablet for 3 replicas of tablet 30fc3de2312f4878863da6eb4304f7d1
I20260810 04:16:37.677073 24415 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fc3de2312f4878863da6eb4304f7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:55456
I20260810 04:16:37.677124 24560 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fc3de2312f4878863da6eb4304f7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:34364
I20260810 04:16:37.677246 24831 tablet_replica.cc:330] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: stopping tablet replica
I20260810 04:16:37.677299 24831 raft_consensus.cc:2229] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.677307 24833 tablet_replica.cc:330] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: stopping tablet replica
I20260810 04:16:37.677336 24831 raft_consensus.cc:2258] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.677380 24833 raft_consensus.cc:2229] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.677428 24833 raft_consensus.cc:2258] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.677552 24708 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fc3de2312f4878863da6eb4304f7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43202
I20260810 04:16:37.677675 24832 tablet_replica.cc:330] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: stopping tablet replica
I20260810 04:16:37.677731 24832 raft_consensus.cc:2229] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:37.677825 24832 raft_consensus.cc:2258] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.677891 24833 ts_tablet_manager.cc:1905] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.679572 24831 ts_tablet_manager.cc:1905] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.680696 24832 ts_tablet_manager.cc:1905] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.681556 24831 ts_tablet_manager.cc:1918] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:37.681689 24831 log.cc:1196] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/30fc3de2312f4878863da6eb4304f7d1
I20260810 04:16:37.681936 24831 ts_tablet_manager.cc:1939] T 30fc3de2312f4878863da6eb4304f7d1 P 2b0e263148fa4edc8103b1664c1d56af: Deleting consensus metadata
I20260810 04:16:37.682554 24832 ts_tablet_manager.cc:1918] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:37.682624 24832 log.cc:1196] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/30fc3de2312f4878863da6eb4304f7d1
I20260810 04:16:37.682857 24832 ts_tablet_manager.cc:1939] T 30fc3de2312f4878863da6eb4304f7d1 P e282c2cd4a8f42378d67a53e805cf61a: Deleting consensus metadata
I20260810 04:16:37.684167 24263 catalog_manager.cc:4841] TS 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007): tablet 30fc3de2312f4878863da6eb4304f7d1 (table test-table [id=17177e82087e49c9b37b832bf3438ab9]) successfully deleted
I20260810 04:16:37.684357 24833 ts_tablet_manager.cc:1918] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:37.684428 24833 log.cc:1196] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/30fc3de2312f4878863da6eb4304f7d1
I20260810 04:16:37.684661 24833 ts_tablet_manager.cc:1939] T 30fc3de2312f4878863da6eb4304f7d1 P c544c740dfc441378335be3f453ff1bf: Deleting consensus metadata
I20260810 04:16:37.685024 24260 catalog_manager.cc:4841] TS c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325): tablet 30fc3de2312f4878863da6eb4304f7d1 (table test-table [id=17177e82087e49c9b37b832bf3438ab9]) successfully deleted
I20260810 04:16:37.685209 24259 catalog_manager.cc:4841] TS e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597): tablet 30fc3de2312f4878863da6eb4304f7d1 (table test-table [id=17177e82087e49c9b37b832bf3438ab9]) successfully deleted
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:16:37.708390 24326 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:37.701439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.15.216.126:40801)
0810 04:16:37.701516 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.701518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.701809 (+ 291us) server_negotiation.cc:407] Connection header received
0810 04:16:37.701876 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.701877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.701887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.701942 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:37.702888 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.703386 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.704453 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.704644 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.704717 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.708307 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38240: BlockingRecv error: recv got EOF from 127.0.0.1:38240 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 04:16:37 dist-test-slave-g350 krb5kdc[24228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM
I20260810 04:16:37.717306 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.709257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.15.216.126:40801)
0810 04:16:37.709298 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.709300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.709352 (+ 52us) server_negotiation.cc:407] Connection header received
0810 04:16:37.709426 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.709426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.709434 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.709480 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:37.710080 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.710590 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.711574 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.711781 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.711879 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.714571 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:37.714573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:37.714573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:37.714588 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:37.716074 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.716489 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.716491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.716491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.716555 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:37.716771 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:37.716773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:37.716773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:37.716895 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:37.717008 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.717109 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.717195 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:16:37.719463 24275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:37.719671 24275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:37.721170 24415 tablet_service.cc:1467] Processing CreateTablet for tablet 02ffec49338649bd83dbb093e486f673 (DEFAULT_TABLE table=test-table [id=51be1cca97c74e5c8f2135926ac64e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.721189 24560 tablet_service.cc:1467] Processing CreateTablet for tablet 02ffec49338649bd83dbb093e486f673 (DEFAULT_TABLE table=test-table [id=51be1cca97c74e5c8f2135926ac64e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.721240 24708 tablet_service.cc:1467] Processing CreateTablet for tablet 02ffec49338649bd83dbb093e486f673 (DEFAULT_TABLE table=test-table [id=51be1cca97c74e5c8f2135926ac64e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.721336 24415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ffec49338649bd83dbb093e486f673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.721342 24560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ffec49338649bd83dbb093e486f673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.721383 24708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ffec49338649bd83dbb093e486f673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.722851 24809 tablet_bootstrap.cc:492] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: Bootstrap starting.
I20260810 04:16:37.722884 24810 tablet_bootstrap.cc:492] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: Bootstrap starting.
I20260810 04:16:37.723261 24809 tablet_bootstrap.cc:654] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.723366 24810 tablet_bootstrap.cc:654] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.723549 24813 tablet_bootstrap.cc:492] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: Bootstrap starting.
I20260810 04:16:37.723915 24809 tablet_bootstrap.cc:492] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: No bootstrap required, opened a new log
I20260810 04:16:37.723973 24809 ts_tablet_manager.cc:1397] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:37.723987 24813 tablet_bootstrap.cc:654] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.724093 24809 raft_consensus.cc:348] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.724170 24809 raft_consensus.cc:374] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.724189 24810 tablet_bootstrap.cc:492] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: No bootstrap required, opened a new log
I20260810 04:16:37.724195 24809 raft_consensus.cc:729] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c544c740dfc441378335be3f453ff1bf, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.724229 24810 ts_tablet_manager.cc:1397] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:37.724252 24809 consensus_queue.cc:260] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.724363 24809 ts_tablet_manager.cc:1428] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.724428 24810 raft_consensus.cc:348] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.724519 24810 raft_consensus.cc:374] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.724546 24810 raft_consensus.cc:729] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0e263148fa4edc8103b1664c1d56af, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.724596 24810 consensus_queue.cc:260] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.724705 24810 ts_tablet_manager.cc:1428] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.725297 24813 tablet_bootstrap.cc:492] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: No bootstrap required, opened a new log
I20260810 04:16:37.725345 24813 ts_tablet_manager.cc:1397] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:16:37.725490 24813 raft_consensus.cc:348] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.725550 24813 raft_consensus.cc:374] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.725569 24813 raft_consensus.cc:729] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282c2cd4a8f42378d67a53e805cf61a, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.725636 24813 consensus_queue.cc:260] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.725721 24813 ts_tablet_manager.cc:1428] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.808238 24815 raft_consensus.cc:482] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:37.808462 24815 raft_consensus.cc:504] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.808884 24815 leader_election.cc:288] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597)
I20260810 04:16:37.812675 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.809369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56889 (local address 127.15.216.65:44325)
0810 04:16:37.809476 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.809478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.809486 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:37.809518 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.809518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.809531 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.809614 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.809893 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.810615 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.811855 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.812248 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.812340 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.812350 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.812522 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.812590 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":52,"threads_started":1}
I20260810 04:16:37.812762 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.809283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:41773 (local address 127.15.216.67:34597)
0810 04:16:37.809360 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.809362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.809449 (+ 87us) server_negotiation.cc:407] Connection header received
0810 04:16:37.809575 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.809576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.809587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.809671 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:16:37.809871 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.810353 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.811800 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.812365 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.812451 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.812461 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.812632 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.812676 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:16:37.812879 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.809065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:44325 (local address 127.15.216.66:56889)
0810 04:16:37.809263 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.809274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.809356 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.809622 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.809623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.809626 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.809708 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.809711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.810715 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.810716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.811767 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.811770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.812339 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.812340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.812449 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.812454 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.812513 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.812564 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260810 04:16:37.813177 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ffec49338649bd83dbb093e486f673" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf" is_pre_election: true
I20260810 04:16:37.813298 24436 raft_consensus.cc:2454] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 0.
I20260810 04:16:37.813506 24511 leader_election.cc:302] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, c544c740dfc441378335be3f453ff1bf; no voters:
I20260810 04:16:37.813634 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.809116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34597 (local address 127.15.216.66:41773)
0810 04:16:37.809392 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.809402 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.809471 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.809724 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.809725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.809728 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:16:37.809807 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.809809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.810454 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.810455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.811671 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.811674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.812473 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.812474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.812541 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.812544 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.813370 (+ 826us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.813464 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":57,"threads_started":1}
I20260810 04:16:37.813714 24815 raft_consensus.cc:2781] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:37.813766 24815 raft_consensus.cc:482] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:37.813781 24815 raft_consensus.cc:3037] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.814216 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ffec49338649bd83dbb093e486f673" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282c2cd4a8f42378d67a53e805cf61a" is_pre_election: true
I20260810 04:16:37.814312 24732 raft_consensus.cc:2454] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 0.
I20260810 04:16:37.814486 24815 raft_consensus.cc:504] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.814646 24815 leader_election.cc:288] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 election: Requested vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597)
I20260810 04:16:37.814916 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ffec49338649bd83dbb093e486f673" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282c2cd4a8f42378d67a53e805cf61a"
I20260810 04:16:37.814994 24732 raft_consensus.cc:3037] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.815013 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ffec49338649bd83dbb093e486f673" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf"
I20260810 04:16:37.815088 24436 raft_consensus.cc:3037] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.815678 24436 raft_consensus.cc:2454] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 1.
I20260810 04:16:37.815845 24511 leader_election.cc:302] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, c544c740dfc441378335be3f453ff1bf; no voters:
I20260810 04:16:37.815934 24732 raft_consensus.cc:2454] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 1.
I20260810 04:16:37.815946 24815 raft_consensus.cc:2781] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:37.816108 24815 raft_consensus.cc:686] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 1 LEADER]: Becoming Leader. State: Replica: 2b0e263148fa4edc8103b1664c1d56af, State: Running, Role: LEADER
I20260810 04:16:37.816200 24815 consensus_queue.cc:237] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.817035 24276 catalog_manager.cc:5495] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66). New cstate: current_term: 1 leader_uuid: "2b0e263148fa4edc8103b1664c1d56af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20260810 04:16:37.822655 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.819852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.15.216.66:33007)
0810 04:16:37.820035 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.820037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.820044 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:37.820076 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.820077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.820094 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.820181 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:37.820375 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.820882 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.821678 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.821940 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.822021 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.822229 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.822390 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.822475 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.822523 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.822547 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
I20260810 04:16:37.823596 24732 raft_consensus.cc:1261] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Refusing update from remote peer 2b0e263148fa4edc8103b1664c1d56af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.823617 24436 raft_consensus.cc:1261] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Refusing update from remote peer 2b0e263148fa4edc8103b1664c1d56af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.823876 24815 consensus_queue.cc:1035] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.824033 24887 consensus_queue.cc:1035] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.825414 24823 mvcc.cc:205] Tried to move back new op lower bound from 7316829789483384832 to 7316829789455622144. Current Snapshot: MvccSnapshot[applied={T|T < 7316829789483384832}]
I20260810 04:16:37.827625 24275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.827726 24275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.828401 24275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Sending DeleteTablet for 3 replicas of tablet 02ffec49338649bd83dbb093e486f673
I20260810 04:16:37.828743 24415 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ffec49338649bd83dbb093e486f673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:55456
I20260810 04:16:37.828763 24708 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ffec49338649bd83dbb093e486f673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43202
I20260810 04:16:37.828871 24833 tablet_replica.cc:330] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: stopping tablet replica
I20260810 04:16:37.828938 24833 raft_consensus.cc:2229] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.828990 24833 raft_consensus.cc:2258] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.829064 24832 tablet_replica.cc:330] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: stopping tablet replica
I20260810 04:16:37.829123 24832 raft_consensus.cc:2229] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.829169 24832 raft_consensus.cc:2258] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.829483 24833 ts_tablet_manager.cc:1905] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.829701 24560 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ffec49338649bd83dbb093e486f673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:34364
I20260810 04:16:37.829964 24275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:16:37.830106 24275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:16:37.830557 24831 tablet_replica.cc:330] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: stopping tablet replica
I20260810 04:16:37.830649 24831 raft_consensus.cc:2229] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:37.830749 24831 raft_consensus.cc:2258] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.831321 24831 ts_tablet_manager.cc:1905] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.831514 24560 tablet_service.cc:1467] Processing CreateTablet for tablet 0e736f3116e448d78382e495441a37f6 (DEFAULT_TABLE table=test-table [id=174d7bb3afc142e482696680c12fe2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.831636 24560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e736f3116e448d78382e495441a37f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.832816 24415 tablet_service.cc:1467] Processing CreateTablet for tablet 0e736f3116e448d78382e495441a37f6 (DEFAULT_TABLE table=test-table [id=174d7bb3afc142e482696680c12fe2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.833009 24810 tablet_bootstrap.cc:492] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: Bootstrap starting.
I20260810 04:16:37.833150 24415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e736f3116e448d78382e495441a37f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.833562 24810 tablet_bootstrap.cc:654] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.834470 24708 tablet_service.cc:1467] Processing CreateTablet for tablet 0e736f3116e448d78382e495441a37f6 (DEFAULT_TABLE table=test-table [id=174d7bb3afc142e482696680c12fe2ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:16:37.834595 24708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e736f3116e448d78382e495441a37f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:37.835269 24833 ts_tablet_manager.cc:1918] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:37.835325 24833 log.cc:1196] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/02ffec49338649bd83dbb093e486f673
I20260810 04:16:37.835556 24833 ts_tablet_manager.cc:1939] T 02ffec49338649bd83dbb093e486f673 P c544c740dfc441378335be3f453ff1bf: Deleting consensus metadata
I20260810 04:16:37.835866 24831 ts_tablet_manager.cc:1918] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:37.835950 24831 log.cc:1196] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/02ffec49338649bd83dbb093e486f673
I20260810 04:16:37.836205 24831 ts_tablet_manager.cc:1939] T 02ffec49338649bd83dbb093e486f673 P 2b0e263148fa4edc8103b1664c1d56af: Deleting consensus metadata
I20260810 04:16:37.836202 24809 tablet_bootstrap.cc:492] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf: Bootstrap starting.
I20260810 04:16:37.836506 24809 tablet_bootstrap.cc:654] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.836601 24263 catalog_manager.cc:4841] TS 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007): tablet 02ffec49338649bd83dbb093e486f673 (table test-table [id=51be1cca97c74e5c8f2135926ac64e40]) successfully deleted
I20260810 04:16:37.836789 24813 tablet_bootstrap.cc:492] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: Bootstrap starting.
I20260810 04:16:37.837194 24813 tablet_bootstrap.cc:654] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:37.837566 24810 tablet_bootstrap.cc:492] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: No bootstrap required, opened a new log
I20260810 04:16:37.837635 24810 ts_tablet_manager.cc:1397] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 04:16:37.837801 24260 catalog_manager.cc:4841] TS c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325): tablet 02ffec49338649bd83dbb093e486f673 (table test-table [id=51be1cca97c74e5c8f2135926ac64e40]) successfully deleted
I20260810 04:16:37.837818 24810 raft_consensus.cc:348] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.837891 24810 raft_consensus.cc:374] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.837913 24810 raft_consensus.cc:729] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0e263148fa4edc8103b1664c1d56af, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.837955 24810 consensus_queue.cc:260] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.838044 24810 ts_tablet_manager.cc:1428] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.838331 24832 ts_tablet_manager.cc:1905] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.839881 24832 ts_tablet_manager.cc:1918] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:16:37.839941 24832 log.cc:1196] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/02ffec49338649bd83dbb093e486f673
I20260810 04:16:37.840176 24832 ts_tablet_manager.cc:1939] T 02ffec49338649bd83dbb093e486f673 P e282c2cd4a8f42378d67a53e805cf61a: Deleting consensus metadata
I20260810 04:16:37.840711 24259 catalog_manager.cc:4841] TS e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597): tablet 02ffec49338649bd83dbb093e486f673 (table test-table [id=51be1cca97c74e5c8f2135926ac64e40]) successfully deleted
I20260810 04:16:37.841060 24809 tablet_bootstrap.cc:492] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf: No bootstrap required, opened a new log
I20260810 04:16:37.841116 24809 ts_tablet_manager.cc:1397] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260810 04:16:37.841173 24813 tablet_bootstrap.cc:492] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: No bootstrap required, opened a new log
I20260810 04:16:37.841216 24813 ts_tablet_manager.cc:1397] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:16:37.841259 24809 raft_consensus.cc:348] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.841332 24809 raft_consensus.cc:374] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.841313 24813 raft_consensus.cc:348] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.841353 24809 raft_consensus.cc:729] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c544c740dfc441378335be3f453ff1bf, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.841372 24813 raft_consensus.cc:374] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:37.841391 24813 raft_consensus.cc:729] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282c2cd4a8f42378d67a53e805cf61a, State: Initialized, Role: FOLLOWER
I20260810 04:16:37.841400 24809 consensus_queue.cc:260] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.841430 24813 consensus_queue.cc:260] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.841497 24809 ts_tablet_manager.cc:1428] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.841509 24813 ts_tablet_manager.cc:1428] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:37.909626 24815 raft_consensus.cc:482] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:16:37.909750 24815 raft_consensus.cc:504] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.909920 24815 leader_election.cc:288] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597)
I20260810 04:16:37.910199 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e736f3116e448d78382e495441a37f6" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282c2cd4a8f42378d67a53e805cf61a" is_pre_election: true
I20260810 04:16:37.910167 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e736f3116e448d78382e495441a37f6" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf" is_pre_election: true
I20260810 04:16:37.910310 24732 raft_consensus.cc:2454] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 0.
I20260810 04:16:37.910337 24436 raft_consensus.cc:2454] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 0.
I20260810 04:16:37.910532 24510 leader_election.cc:302] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, e282c2cd4a8f42378d67a53e805cf61a; no voters:
I20260810 04:16:37.910681 24815 raft_consensus.cc:2781] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:16:37.910719 24815 raft_consensus.cc:482] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:16:37.910743 24815 raft_consensus.cc:3037] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.911849 24815 raft_consensus.cc:504] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.912035 24815 leader_election.cc:288] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 election: Requested vote from peers c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325), e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597)
I20260810 04:16:37.912202 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e736f3116e448d78382e495441a37f6" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c544c740dfc441378335be3f453ff1bf"
I20260810 04:16:37.912323 24436 raft_consensus.cc:3037] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.912305 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e736f3116e448d78382e495441a37f6" candidate_uuid: "2b0e263148fa4edc8103b1664c1d56af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e282c2cd4a8f42378d67a53e805cf61a"
I20260810 04:16:37.912400 24732 raft_consensus.cc:3037] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:37.913002 24436 raft_consensus.cc:2454] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 1.
I20260810 04:16:37.913208 24511 leader_election.cc:302] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0e263148fa4edc8103b1664c1d56af, c544c740dfc441378335be3f453ff1bf; no voters:
I20260810 04:16:37.913306 24732 raft_consensus.cc:2454] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0e263148fa4edc8103b1664c1d56af in term 1.
I20260810 04:16:37.913335 24815 raft_consensus.cc:2781] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:37.913381 24815 raft_consensus.cc:686] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 1 LEADER]: Becoming Leader. State: Replica: 2b0e263148fa4edc8103b1664c1d56af, State: Running, Role: LEADER
I20260810 04:16:37.913436 24815 consensus_queue.cc:237] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } }
I20260810 04:16:37.914093 24276 catalog_manager.cc:5495] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66). New cstate: current_term: 1 leader_uuid: "2b0e263148fa4edc8103b1664c1d56af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0e263148fa4edc8103b1664c1d56af" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 33007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20260810 04:16:37.927690 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.925115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:41433 (local address 127.15.216.67:34597)
0810 04:16:37.925177 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.925179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.925327 (+ 148us) server_negotiation.cc:407] Connection header received
0810 04:16:37.925402 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.925402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.925415 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.925472 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:37.925648 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.926145 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.926911 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.927120 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.927195 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.927402 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.927442 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.927502 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.927565 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.927588 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:16:37.927753 24902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.925015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34597 (local address 127.15.216.66:41433)
0810 04:16:37.925218 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.925229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.925346 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.925476 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.925477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.925478 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:37.925552 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.925554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.926230 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.926231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.926820 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.926822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.927206 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.927207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.927300 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.927315 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.927505 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.927507 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.927561 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.927651 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260810 04:16:37.930112 24436 raft_consensus.cc:1261] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Refusing update from remote peer 2b0e263148fa4edc8103b1664c1d56af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.930125 24732 raft_consensus.cc:1261] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Refusing update from remote peer 2b0e263148fa4edc8103b1664c1d56af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:16:37.930416 24894 consensus_queue.cc:1035] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e282c2cd4a8f42378d67a53e805cf61a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.930526 24815 consensus_queue.cc:1035] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c544c740dfc441378335be3f453ff1bf" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:16:37.938515 24860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.935481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:33007 (local address 127.15.216.67:35489)
0810 04:16:37.935551 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.935564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:37.935607 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:37.935787 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:37.935789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:37.935790 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 04:16:37.935841 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.935842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.936645 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.936646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.937506 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:37.937509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.937978 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.937979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:37.938042 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.938050 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.938347 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.938349 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:37.938394 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 04:16:37.938456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 04:16:37.938740 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.935543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:35489 (local address 127.15.216.66:33007)
0810 04:16:37.935661 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:37.935663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:37.935672 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:37.935696 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:37.935697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:37.935708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:37.935764 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:16:37.935978 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.936560 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.937669 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:37.937900 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:37.937977 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:37.938166 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:16:37.938204 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:16:37.938285 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:37.938640 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 04:16:37.938673 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 04:16:37.952481 24275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.952615 24275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:16:37.953373 24275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb158e903ed44f9a08a72b85960c17c: Sending DeleteTablet for 3 replicas of tablet 0e736f3116e448d78382e495441a37f6
I20260810 04:16:37.953675 24708 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e736f3116e448d78382e495441a37f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43202
I20260810 04:16:37.953696 24560 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e736f3116e448d78382e495441a37f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:34364
I20260810 04:16:37.953794 24832 tablet_replica.cc:330] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: stopping tablet replica
I20260810 04:16:37.953833 24831 tablet_replica.cc:330] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: stopping tablet replica
I20260810 04:16:37.953853 24832 raft_consensus.cc:2229] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.953899 24832 raft_consensus.cc:2258] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.953891 24831 raft_consensus.cc:2229] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:16:37.953949 24831 raft_consensus.cc:2258] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.954216 24415 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e736f3116e448d78382e495441a37f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:16:37 UTC) from {username='oryx', principal='oryx/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:55456
I20260810 04:16:37.954315 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 24350
I20260810 04:16:37.954384 24833 tablet_replica.cc:330] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf: stopping tablet replica
I20260810 04:16:37.954391 24832 ts_tablet_manager.cc:1905] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.954433 24833 raft_consensus.cc:2229] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:16:37.954464 24833 raft_consensus.cc:2258] T 0e736f3116e448d78382e495441a37f6 P c544c740dfc441378335be3f453ff1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:16:37.957500 24832 ts_tablet_manager.cc:1918] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:37.957563 24832 log.cc:1196] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/0e736f3116e448d78382e495441a37f6
I20260810 04:16:37.957810 24832 ts_tablet_manager.cc:1939] T 0e736f3116e448d78382e495441a37f6 P e282c2cd4a8f42378d67a53e805cf61a: Deleting consensus metadata
I20260810 04:16:37.958307 24831 ts_tablet_manager.cc:1905] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:16:37.959451 24259 catalog_manager.cc:4841] TS e282c2cd4a8f42378d67a53e805cf61a (127.15.216.67:34597): tablet 0e736f3116e448d78382e495441a37f6 (table test-table [id=174d7bb3afc142e482696680c12fe2ec]) successfully deleted
I20260810 04:16:37.959827 24831 ts_tablet_manager.cc:1918] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 04:16:37.959894 24831 log.cc:1196] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/0e736f3116e448d78382e495441a37f6
I20260810 04:16:37.960099 24831 ts_tablet_manager.cc:1939] T 0e736f3116e448d78382e495441a37f6 P 2b0e263148fa4edc8103b1664c1d56af: Deleting consensus metadata
I20260810 04:16:37.960570 24263 catalog_manager.cc:4841] TS 2b0e263148fa4edc8103b1664c1d56af (127.15.216.66:33007): tablet 0e736f3116e448d78382e495441a37f6 (table test-table [id=174d7bb3afc142e482696680c12fe2ec]) successfully deleted
W20260810 04:16:37.966156 24260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:44325 (error 108)
I20260810 04:16:37.966430 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 24494
I20260810 04:16:37.967773 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:37.966616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:44325 (local address 127.0.0.1:55472)
0810 04:16:37.967677 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:37.967728 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":39,"threads_started":1}
W20260810 04:16:37.967875 24260 catalog_manager.cc:4584] TS c544c740dfc441378335be3f453ff1bf (127.15.216.65:44325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e736f3116e448d78382e495441a37f6: Network error: Client connection negotiation failed: client connection to 127.15.216.65:44325: connect: Connection refused (error 111)
I20260810 04:16:37.975119 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 24640
I20260810 04:16:37.985433 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 24241
2026-08-10T04:16:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2858 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:38 dist-test-slave-g350 krb5kdc[24909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:38 dist-test-slave-g350 krb5kdc[24909](info): set up 2 sockets
Aug 10 04:16:38 dist-test-slave-g350 krb5kdc[24909](info): commencing operation
krb5kdc: starting...
W20260810 04:16:39.092577 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:39 dist-test-slave-g350 krb5kdc[24909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:39.234128 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:46727
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:39.321933 24923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:39.322197 24923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:39.322238 24923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:39.323472 24923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:39.323549 24923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:39.323575 24923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:39.323606 24923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:39.323632 24923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:39.325183 24923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:39.325502 24923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:39.325778 24923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:39.331240 24929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:39.331228 24930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:39.331364 24923 server_base.cc:1034] running on GCE node
W20260810 04:16:39.331240 24932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:39.331954 24923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:39.332311 24923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:39.333510 24923 hybrid_clock.cc:648] HybridClock initialized: now 1786335399333496 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T04:16:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1354 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:39 dist-test-slave-g350 krb5kdc[24937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:39 dist-test-slave-g350 krb5kdc[24937](info): set up 2 sockets
Aug 10 04:16:39 dist-test-slave-g350 krb5kdc[24937](info): commencing operation
krb5kdc: starting...
W20260810 04:16:40.457017 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:40 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:40.596889 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39089
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:39089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:40.683282 24950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:40.683516 24950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:40.683540 24950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:40.684803 24950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:40.684854 24950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:40.684870 24950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:40.684882 24950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:40.684895 24950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:40.686529 24950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:39089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:40.686826 24950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:40.687068 24950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:40.692502 24956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:40.692510 24958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:40.692512 24955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:40.692678 24950 server_base.cc:1034] running on GCE node
I20260810 04:16:40.693064 24950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:40.693426 24950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:40.694604 24950 hybrid_clock.cc:648] HybridClock initialized: now 1786335400694600 us; error 58 us; skew 500 ppm
Aug 10 04:16:40 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335400, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:40.701818 24950 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:40.702399 24950 webserver.cc:460] Webserver started at http://127.15.216.126:43681/ using document root <none> and password file <none>
I20260810 04:16:40.702651 24950 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:40.702724 24950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:40.702884 24950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:40.704023 24950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "820e329396db49749eb092be728386ff"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "f473b4f29dd1882f648e95c9ee44f07f"
server_key_iv: "e868a1a4c159e67d56132fd1c54433df"
server_key_version: "encryptionkey@0"
I20260810 04:16:40.704414 24950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "820e329396db49749eb092be728386ff"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "f473b4f29dd1882f648e95c9ee44f07f"
server_key_iv: "e868a1a4c159e67d56132fd1c54433df"
server_key_version: "encryptionkey@0"
I20260810 04:16:40.705894 24950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:40.706763 24965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:40.706969 24950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:40.707093 24950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "820e329396db49749eb092be728386ff"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "f473b4f29dd1882f648e95c9ee44f07f"
server_key_iv: "e868a1a4c159e67d56132fd1c54433df"
server_key_version: "encryptionkey@0"
I20260810 04:16:40.707175 24950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:40.726647 24950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:40.728956 24950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:40.729144 24950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:40.733803 24950 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:39089
I20260810 04:16:40.733856 25019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:39089 every 8 connection(s)
I20260810 04:16:40.734243 24950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:40.735137 25020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:40.737720 25020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Bootstrap starting.
I20260810 04:16:40.738404 25020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:40.738724 25020 log.cc:824] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:40.739585 25020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: No bootstrap required, opened a new log
I20260810 04:16:40.742100 25020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:40.742260 25020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:40.742312 25020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e329396db49749eb092be728386ff, State: Initialized, Role: FOLLOWER
I20260810 04:16:40.742476 25020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:40.742547 25020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:40.742591 25020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:40.742645 25020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:40.743296 25020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:40.743398 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 24950
I20260810 04:16:40.743423 25020 leader_election.cc:302] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 820e329396db49749eb092be728386ff; no voters:
I20260810 04:16:40.743516 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:40.743634 25020 leader_election.cc:288] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:40.743693 25024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:40.743760 16225 external_mini_cluster.cc:1428] Setting key de599ed8b7fba2054ea4bfe3c46eda55
I20260810 04:16:40.743863 25024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 LEADER]: Becoming Leader. State: Replica: 820e329396db49749eb092be728386ff, State: Running, Role: LEADER
I20260810 04:16:40.743934 25020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:40.744050 25024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:40.744469 25026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "820e329396db49749eb092be728386ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } } }
I20260810 04:16:40.744556 25026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:40.744807 25027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 820e329396db49749eb092be728386ff. Latest consensus state: current_term: 1 leader_uuid: "820e329396db49749eb092be728386ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } } }
I20260810 04:16:40.744944 25027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:40.745465 25032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:40.746133 25032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:40.747920 25032 catalog_manager.cc:1347] Generated new cluster ID: a5569223292841d89e1263ea74266e77
I20260810 04:16:40.747975 25032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:16:40 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:40.755285 25032 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:40.756398 25032 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:40.759521 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:40.744735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.15.216.126:39089)
0810 04:16:40.744959 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:40.744970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:40.745006 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:16:40.745155 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:40.745164 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:40.745228 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:40.746273 (+ 1045us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:40.746367 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:40.747656 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:40.748556 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:40.748983 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:40.749110 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:40.755078 (+ 5968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:40.755083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:40.755083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:40.755098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:40.757461 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:40.757992 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:40.757997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:40.757998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:40.758117 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:40.758373 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:40.758374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:40.758374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:40.758668 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:40.758834 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:40.758964 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 04:16:40.759259 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260810 04:16:40.766124 25032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Generated new TSK 0
I20260810 04:16:40.766368 25032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:40.801481 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39089
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:40.909300 25048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:40.909613 25048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:40.909642 25048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:40.910959 25048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:40.911005 25048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:40.911032 25048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:40.912569 25048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:40.912837 25048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:40.913082 25048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:40.918488 25056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:40.918497 25054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:40.918509 25053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:40.918838 25048 server_base.cc:1034] running on GCE node
I20260810 04:16:40.919145 25048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:40.919457 25048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:40.920639 25048 hybrid_clock.cc:648] HybridClock initialized: now 1786335400920623 us; error 48 us; skew 500 ppm
Aug 10 04:16:40 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335400, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:40.923477 25048 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:40.924037 25048 webserver.cc:460] Webserver started at http://127.15.216.65:35969/ using document root <none> and password file <none>
I20260810 04:16:40.924299 25048 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:40.924373 25048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:40.924536 25048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:40.925768 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "78efa980f5904fa0a14a26d582fb5c9d"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "0b1c159173bad346223a37ccf51fb7e4"
server_key_iv: "a4da6c06fc50f952f93b0c391844237b"
server_key_version: "encryptionkey@0"
I20260810 04:16:40.926167 25048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "78efa980f5904fa0a14a26d582fb5c9d"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "0b1c159173bad346223a37ccf51fb7e4"
server_key_iv: "a4da6c06fc50f952f93b0c391844237b"
server_key_version: "encryptionkey@0"
I20260810 04:16:40.927661 25048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:40.928733 25064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:40.928977 25048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:40.929098 25048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "78efa980f5904fa0a14a26d582fb5c9d"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "0b1c159173bad346223a37ccf51fb7e4"
server_key_iv: "a4da6c06fc50f952f93b0c391844237b"
server_key_version: "encryptionkey@0"
I20260810 04:16:40.929229 25048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:40.941658 25048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:40.944334 25048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:40.944511 25048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:40.944828 25048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:40.945257 25048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:40.945322 25048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:40.945367 25048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:40.945396 25048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:40.952099 25048 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:44711
I20260810 04:16:40.952158 25187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:44711 every 8 connection(s)
I20260810 04:16:40.952510 25048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:40.958691 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 25048
I20260810 04:16:40.958882 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:40.959137 16225 external_mini_cluster.cc:1428] Setting key 21363fbb5990f96c08101de6df359dce
Aug 10 04:16:40 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335400, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:40.964285 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:40.953313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:46397 (local address 127.15.216.126:39089)
0810 04:16:40.953401 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:40.953404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:40.953876 (+ 472us) server_negotiation.cc:407] Connection header received
0810 04:16:40.954564 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:40.954566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:40.954578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:40.954644 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:40.955742 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:40.956473 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:40.957513 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:40.957813 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:40.957876 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:40.960725 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:40.960727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:40.960728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:40.960737 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:40.962441 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:40.963034 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:40.963035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:40.963036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:40.963101 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:40.963460 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:40.963461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:40.963461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:40.963581 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:40.963641 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:40.964096 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 04:16:40.964208 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:16:40.964668 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:40.953353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:46397)
0810 04:16:40.953635 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:40.953695 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:40.954271 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:40.954753 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:40.954756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:40.955392 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:40.955605 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:40.955609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:40.956578 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:40.956579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:40.957420 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:40.957423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:40.957911 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:40.957912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:40.958029 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:40.958652 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:40.958666 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:40.960618 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:40.962543 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:40.962554 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:40.962557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:40.962943 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:40.963180 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:40.963181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:40.963181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:40.963375 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:40.963657 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:40.963665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:40.963870 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 04:16:40.964007 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 04:16:40.964251 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
I20260810 04:16:40.965427 25188 heartbeater.cc:347] Connected to a master server at 127.15.216.126:39089
I20260810 04:16:40.965562 25188 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:40.965831 25188 heartbeater.cc:511] Master 127.15.216.126:39089 requested a full tablet report, sending...
I20260810 04:16:40.966531 24983 ts_manager.cc:194] Registered new tserver with Master: 78efa980f5904fa0a14a26d582fb5c9d (127.15.216.65:44711)
I20260810 04:16:40.967176 24983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:46397
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:40.993148 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39089
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:41.079342 25194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:41.079562 25194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:41.079588 25194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:41.080813 25194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:41.080860 25194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:41.080888 25194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:41.082454 25194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:41.082732 25194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:41.082988 25194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:41.088634 25200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:41.088650 25194 server_base.cc:1034] running on GCE node
W20260810 04:16:41.088604 25199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.088613 25202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:41.089237 25194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:41.089459 25194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:41.090610 25194 hybrid_clock.cc:648] HybridClock initialized: now 1786335401090611 us; error 55 us; skew 500 ppm
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:41.093312 25194 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:16:41.093855 25194 webserver.cc:460] Webserver started at http://127.15.216.66:44459/ using document root <none> and password file <none>
I20260810 04:16:41.094111 25194 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:41.094180 25194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:41.094338 25194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:41.095420 25194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "11fab6e26da74b908dec8e412a5b9d2a"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "e215c42133925c8079c83ed255f02cf5"
server_key_iv: "0ad6fb8ba1f6d4a2c113a1cbf1f19991"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.095784 25194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "11fab6e26da74b908dec8e412a5b9d2a"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "e215c42133925c8079c83ed255f02cf5"
server_key_iv: "0ad6fb8ba1f6d4a2c113a1cbf1f19991"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.097180 25194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:41.098098 25210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.098327 25194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:16:41.098477 25194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "11fab6e26da74b908dec8e412a5b9d2a"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "e215c42133925c8079c83ed255f02cf5"
server_key_iv: "0ad6fb8ba1f6d4a2c113a1cbf1f19991"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.098558 25194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:41.116969 25194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:41.119181 25194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:41.119352 25194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:41.119638 25194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:41.119971 25194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:41.120035 25194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.120076 25194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:41.120121 25194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.126451 25194 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:38597
I20260810 04:16:41.126500 25329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:38597 every 8 connection(s)
I20260810 04:16:41.126866 25194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:41.129742 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 25194
I20260810 04:16:41.129868 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:41.130069 16225 external_mini_cluster.cc:1428] Setting key c83fee0b19b876aa53e214f87fda06df
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:41.137941 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.127732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56923 (local address 127.15.216.126:39089)
0810 04:16:41.127781 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.127783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.127977 (+ 194us) server_negotiation.cc:407] Connection header received
0810 04:16:41.128487 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.128488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.128499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.128554 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.129506 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.130180 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.131583 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.131768 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.131833 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.134599 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:41.134601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:41.134601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:41.134611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:41.136379 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:41.136787 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:41.136788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:41.136789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:41.136867 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:41.137111 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:41.137112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:41.137113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:41.137289 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:41.137371 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:41.137679 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 04:16:41.137781 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:16:41.138010 25332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.127697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:56923)
0810 04:16:41.127864 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.127906 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.128381 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.128608 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.128611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.129204 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.129406 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.129411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.130270 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.130271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.131499 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.131503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.131860 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.131861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.132017 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.132611 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.132621 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.134495 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:41.136471 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:41.136478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:41.136480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:41.136703 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:41.136943 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:41.136944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:41.136944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:41.137042 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:41.137361 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:41.137366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:41.137513 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 04:16:41.137661 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 04:16:41.137798 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":107,"threads_started":1}
I20260810 04:16:41.138587 25330 heartbeater.cc:347] Connected to a master server at 127.15.216.126:39089
I20260810 04:16:41.138703 25330 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:41.138903 25330 heartbeater.cc:511] Master 127.15.216.126:39089 requested a full tablet report, sending...
I20260810 04:16:41.139343 24983 ts_manager.cc:194] Registered new tserver with Master: 11fab6e26da74b908dec8e412a5b9d2a (127.15.216.66:38597)
I20260810 04:16:41.139927 24983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:56923
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:16:41.161964 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39089
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:41.248358 25335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:41.248607 25335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:41.248651 25335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:41.249898 25335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:41.249969 25335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:41.250016 25335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:41.251597 25335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:41.251892 25335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:41.252138 25335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:41.257582 25343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.257591 25341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.257582 25340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:41.258224 25335 server_base.cc:1034] running on GCE node
I20260810 04:16:41.258522 25335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:41.258802 25335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:41.259989 25335 hybrid_clock.cc:648] HybridClock initialized: now 1786335401259991 us; error 47 us; skew 500 ppm
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:41.262755 25335 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:16:41.263230 25335 webserver.cc:460] Webserver started at http://127.15.216.67:44685/ using document root <none> and password file <none>
I20260810 04:16:41.263487 25335 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:41.263559 25335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:41.263743 25335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:41.264967 25335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "09ad3ba64fc542689f1c1d10281f51bb"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "94093ca3cda3b5bb3e61fe1eb1aa44ab"
server_key_iv: "17fc232a1f0f83c278a0241cfbc49285"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.265341 25335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "09ad3ba64fc542689f1c1d10281f51bb"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "94093ca3cda3b5bb3e61fe1eb1aa44ab"
server_key_iv: "17fc232a1f0f83c278a0241cfbc49285"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.266963 25335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:41.267859 25350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.268118 25335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:41.268244 25335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "09ad3ba64fc542689f1c1d10281f51bb"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "94093ca3cda3b5bb3e61fe1eb1aa44ab"
server_key_iv: "17fc232a1f0f83c278a0241cfbc49285"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.268359 25335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:41.278285 25335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:41.280695 25335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:41.280894 25335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:41.281167 25335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:41.281508 25335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:41.281572 25335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.281631 25335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:41.281680 25335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.288168 25335 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:35175
I20260810 04:16:41.288220 25470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:35175 every 8 connection(s)
I20260810 04:16:41.288616 25335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:41.298692 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 25335
I20260810 04:16:41.298856 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:41.299014 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.289394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:34751 (local address 127.15.216.126:39089)
0810 04:16:41.289459 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.289461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.289755 (+ 294us) server_negotiation.cc:407] Connection header received
0810 04:16:41.290319 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.290320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.290334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.290380 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.291373 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.291884 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.292911 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.293128 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.293208 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.295612 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:41.295614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:41.295614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:41.295621 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:41.297292 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:41.297756 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:41.297757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:41.297758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:41.297833 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:41.298150 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:41.298151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:41.298151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:41.298345 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:41.298409 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:41.298816 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 04:16:41.298918 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:16:41.299082 16225 external_mini_cluster.cc:1428] Setting key be231689e7899f91144bd4349b806e81
I20260810 04:16:41.299204 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.289395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:34751)
0810 04:16:41.289574 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.289659 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.290207 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.290449 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.290453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.291052 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.291268 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.291272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.291996 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.291997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.292814 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.292816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.293234 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.293234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.293369 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.293921 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.293929 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.295512 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:41.297409 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:41.297417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:41.297419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:41.297676 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:41.297944 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:41.297944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:41.297945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:41.298067 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:41.298453 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:41.298460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:41.298594 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 04:16:41.298740 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 04:16:41.298940 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260810 04:16:41.299775 25471 heartbeater.cc:347] Connected to a master server at 127.15.216.126:39089
I20260810 04:16:41.299916 25471 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:41.300127 25471 heartbeater.cc:511] Master 127.15.216.126:39089 requested a full tablet report, sending...
I20260810 04:16:41.300555 24982 ts_manager.cc:194] Registered new tserver with Master: 09ad3ba64fc542689f1c1d10281f51bb (127.15.216.67:35175)
I20260810 04:16:41.301020 24982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:34751
I20260810 04:16:41.311157 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab.
Entry for principal oryx/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab.
WARNING: no policy specified for oryx/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab.
Entry for principal oryx/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab.
WARNING: no policy specified for oryx/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab.
Entry for principal oryx/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab.
I20260810 04:16:41.401252 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 25048
I20260810 04:16:41.408367 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 25194
I20260810 04:16:41.416894 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 25335
I20260810 04:16:41.423833 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 24950
I20260810 04:16:41.431398 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39089
--webserver_interface=127.15.216.126
--webserver_port=43681
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:39089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:41.519346 25481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:41.519584 25481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:41.519611 25481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:41.520891 25481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:41.520943 25481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:41.520959 25481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:41.520973 25481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:41.520987 25481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:41.522748 25481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:39089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=43681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:41.523036 25481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:41.523298 25481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:16:41.529057 25481 server_base.cc:1034] running on GCE node
W20260810 04:16:41.528999 25489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.528999 25487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.529031 25486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:41.529755 25481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:41.530097 25481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:41.531294 25481 hybrid_clock.cc:648] HybridClock initialized: now 1786335401531282 us; error 30 us; skew 500 ppm
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:41.534340 25481 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:41.534771 25481 webserver.cc:460] Webserver started at http://127.15.216.126:43681/ using document root <none> and password file <none>
I20260810 04:16:41.534981 25481 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:41.535035 25481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:41.536582 25481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:41.537439 25496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.537686 25481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:41.537794 25481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "820e329396db49749eb092be728386ff"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "f473b4f29dd1882f648e95c9ee44f07f"
server_key_iv: "e868a1a4c159e67d56132fd1c54433df"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.538168 25481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:41.556890 25481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:41.559473 25481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:41.559652 25481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:41.564121 25481 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:39089
I20260810 04:16:41.564172 25551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:39089 every 8 connection(s)
I20260810 04:16:41.564646 25481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:41.566270 25552 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:16:41.566890 25552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Bootstrap starting.
I20260810 04:16:41.567283 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 25481
I20260810 04:16:41.567595 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:44711
--local_ip_for_outbound_sockets=127.15.216.65
--tserver_master_addrs=127.15.216.126:39089
--webserver_port=35969
--webserver_interface=127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:16:41.577118 25552 log.cc:824] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:41.579092 25552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:16:41.579457 25552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Bootstrap complete.
I20260810 04:16:41.582093 25552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:41.582360 25552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820e329396db49749eb092be728386ff, State: Initialized, Role: FOLLOWER
I20260810 04:16:41.582554 25552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:41.582625 25552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:41.582669 25552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:41.582715 25552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:16:41.583815 25552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:41.583914 25552 leader_election.cc:302] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 820e329396db49749eb092be728386ff; no voters:
I20260810 04:16:41.584146 25552 leader_election.cc:288] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:16:41.584190 25556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:16:41.584357 25556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [term 2 LEADER]: Becoming Leader. State: Replica: 820e329396db49749eb092be728386ff, State: Running, Role: LEADER
I20260810 04:16:41.584463 25556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } }
I20260810 04:16:41.584503 25552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:41.584728 25556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 820e329396db49749eb092be728386ff. Latest consensus state: current_term: 2 leader_uuid: "820e329396db49749eb092be728386ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } } }
I20260810 04:16:41.584820 25556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:41.584733 25557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "820e329396db49749eb092be728386ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820e329396db49749eb092be728386ff" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39089 } } }
I20260810 04:16:41.584986 25557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:41.585031 25561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:41.585615 25561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:41.585799 25561 catalog_manager.cc:1259] Loaded cluster ID: a5569223292841d89e1263ea74266e77
I20260810 04:16:41.586082 25561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:41.586913 25561 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:41.587360 25561 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 820e329396db49749eb092be728386ff: Loaded TSK: 0
I20260810 04:16:41.587589 25561 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:16:41.665418 25554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:41.665743 25554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:41.665791 25554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:41.667029 25554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:41.667105 25554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:41.667153 25554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:41.668717 25554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.65.keytab
--principal=oryx/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:44711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=35969
--tserver_master_addrs=127.15.216.126:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:41.669009 25554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:41.669286 25554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:16:41.675028 25554 server_base.cc:1034] running on GCE node
W20260810 04:16:41.675076 25579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.675086 25580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.675166 25582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:41.675633 25554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:41.675887 25554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:41.677083 25554 hybrid_clock.cc:648] HybridClock initialized: now 1786335401677067 us; error 34 us; skew 500 ppm
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:41.680006 25554 init.cc:377] Logged in from keytab as oryx/127.15.216.65@KRBTEST.COM (short username oryx)
I20260810 04:16:41.680482 25554 webserver.cc:460] Webserver started at http://127.15.216.65:35969/ using document root <none> and password file <none>
I20260810 04:16:41.680729 25554 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:41.680820 25554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:41.682533 25554 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:41.683317 25590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.683539 25554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:41.683656 25554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "78efa980f5904fa0a14a26d582fb5c9d"
format_stamp: "Formatted at 2026-08-10 04:16:40 on dist-test-slave-g350"
server_key: "0b1c159173bad346223a37ccf51fb7e4"
server_key_iv: "a4da6c06fc50f952f93b0c391844237b"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.684073 25554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:41.717938 25554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:41.720450 25554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:41.720604 25554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:41.720901 25554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:41.721303 25554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:41.721362 25554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.721405 25554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:41.721433 25554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.728189 25554 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:44711
I20260810 04:16:41.728312 25709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:44711 every 8 connection(s)
I20260810 04:16:41.728731 25554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:41.734961 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 25554
I20260810 04:16:41.735421 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:38597
--local_ip_for_outbound_sockets=127.15.216.66
--tserver_master_addrs=127.15.216.126:39089
--webserver_port=44459
--webserver_interface=127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:41.738471 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.729908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:56151)
0810 04:16:41.730101 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.730146 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.730653 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.731078 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.731081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.731672 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.731889 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.731893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.733170 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.733171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.734137 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.734140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.734757 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.734758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.734866 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.735580 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.735592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.738302 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":106,"threads_started":1}
W20260810 04:16:41.738660 25713 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:41.729960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:56151 (local address 127.15.216.126:39089)
0810 04:16:41.730147 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.730156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.730238 (+ 82us) server_negotiation.cc:407] Connection header received
0810 04:16:41.730851 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.730856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.730899 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.731017 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.732022 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.733052 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.734222 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.734640 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.734769 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.738488 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:56151: BlockingRecv error: recv got EOF from 127.15.216.65:56151 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
W20260810 04:16:41.738742 25710 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:39089 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.216.126:39089: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:41.744131 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.738999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:41775)
0810 04:16:41.739053 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.739064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.739113 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.739400 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.739401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.739546 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.739595 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.739596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.740364 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.740365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.740871 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.740872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.741296 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.741297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.741372 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.741908 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.741913 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.744025 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:16:41.744309 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:41.739069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:41775 (local address 127.15.216.126:39089)
0810 04:16:41.739270 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.739272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.739279 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:16:41.739301 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.739301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.739311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.739393 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.739675 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.740273 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.740956 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.741224 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.741298 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.744128 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:41775: BlockingRecv error: recv got EOF from 127.15.216.65:41775 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:41.749176 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.744498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:41551)
0810 04:16:41.744558 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.744792 (+ 234us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.744868 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.745048 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.745049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.745186 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.745219 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.745221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.745968 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.745969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.746471 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.746472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.746833 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.746834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.746889 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.747345 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.747350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.749083 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 04:16:41.749231 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:41.744569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:41551 (local address 127.15.216.126:39089)
0810 04:16:41.744676 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.744678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.744852 (+ 174us) server_negotiation.cc:407] Connection header received
0810 04:16:41.744957 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.744959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.744970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.745043 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.745310 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.745869 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.746559 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.746757 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.746835 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.749165 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:41551: BlockingRecv error: recv got EOF from 127.15.216.65:41551 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":107,"threads_started":1}
W20260810 04:16:41.749332 25710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:16:41.824416 25714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:41.824651 25714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:41.824676 25714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:41.826025 25714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:41.826073 25714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:41.826103 25714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:41.828008 25714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.66.keytab
--principal=oryx/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:38597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=44459
--tserver_master_addrs=127.15.216.126:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:41.828295 25714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:41.828547 25714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:41.834447 25722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.834478 25721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.834554 25724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:41.834649 25714 server_base.cc:1034] running on GCE node
I20260810 04:16:41.834869 25714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:41.835158 25714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:41.836342 25714 hybrid_clock.cc:648] HybridClock initialized: now 1786335401836289 us; error 86 us; skew 500 ppm
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:41.839183 25714 init.cc:377] Logged in from keytab as oryx/127.15.216.66@KRBTEST.COM (short username oryx)
I20260810 04:16:41.839651 25714 webserver.cc:460] Webserver started at http://127.15.216.66:44459/ using document root <none> and password file <none>
I20260810 04:16:41.839907 25714 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:41.840005 25714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:41.841673 25714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:41.842525 25731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.842792 25714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:41.842918 25714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "11fab6e26da74b908dec8e412a5b9d2a"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "e215c42133925c8079c83ed255f02cf5"
server_key_iv: "0ad6fb8ba1f6d4a2c113a1cbf1f19991"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.843375 25714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:41.855895 25714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:41.858577 25714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:41.858771 25714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:41.859064 25714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:41.859431 25714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:41.859493 25714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.859562 25714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:41.859594 25714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.866458 25714 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:38597
I20260810 04:16:41.866521 25849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:38597 every 8 connection(s)
I20260810 04:16:41.866914 25714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:41.872843 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 25714
I20260810 04:16:41.873189 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:35175
--local_ip_for_outbound_sockets=127.15.216.67
--tserver_master_addrs=127.15.216.126:39089
--webserver_port=44685
--webserver_interface=127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:41.875823 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.867759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:56733)
0810 04:16:41.867990 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.868031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.868546 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.868812 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.868815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.869454 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.869775 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.869781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.870599 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.870600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.871429 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.871432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.871892 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.871893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.871998 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.872696 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.872706 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.875645 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":134,"threads_started":1}
W20260810 04:16:41.875859 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:41.867809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56733 (local address 127.15.216.126:39089)
0810 04:16:41.867913 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.867915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.868139 (+ 224us) server_negotiation.cc:407] Connection header received
0810 04:16:41.868639 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.868640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.868656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.868713 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.869902 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.870471 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.871539 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.871797 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.871872 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.875777 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:56733: BlockingRecv error: recv got EOF from 127.15.216.66:56733 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 04:16:41.876118 25850 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:39089 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.216.126:39089: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:41.882112 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.876418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:49227)
0810 04:16:41.876460 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.876549 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.876609 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.876744 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.876744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.876910 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.876973 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.876975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.877718 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.877719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.878628 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.878632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.879027 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.879029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.879112 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.879674 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.879681 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.882007 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 04:16:41.882177 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:41.876462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:49227 (local address 127.15.216.126:39089)
0810 04:16:41.876507 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.876510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.876591 (+ 81us) server_negotiation.cc:407] Connection header received
0810 04:16:41.876671 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.876672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.876684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.876730 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.877057 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.877577 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.878726 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.878945 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.879007 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.882107 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:49227: BlockingRecv error: recv got EOF from 127.15.216.66:49227 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:41.887866 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:41.882420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:42263)
0810 04:16:41.882569 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:41.882581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:41.882623 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:41.882805 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:41.882806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:41.882974 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:41.883019 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.883021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.883685 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.883686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.884631 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:41.884634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.885057 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.885058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:41.885127 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.885666 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:41.885671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:41.887782 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
W20260810 04:16:41.887976 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:41.882549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:42263 (local address 127.15.216.126:39089)
0810 04:16:41.882609 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:41.882610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:41.882626 (+ 16us) server_negotiation.cc:407] Connection header received
0810 04:16:41.882714 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:41.882714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:41.882723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:41.882775 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:41.883099 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.883602 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.884782 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:41.884979 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:41.885035 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:41.887881 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:42263: BlockingRecv error: recv got EOF from 127.15.216.66:42263 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 04:16:41.888010 25850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:16:41.967015 25854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:41.967242 25854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:41.967276 25854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:41.968466 25854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:41.968513 25854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:41.968540 25854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:41.970093 25854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/krb5kdc/oryx_127.15.216.67.keytab
--principal=oryx/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:35175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=44685
--tserver_master_addrs=127.15.216.126:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:41.970363 25854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:41.970628 25854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:16:41.976570 25854 server_base.cc:1034] running on GCE node
W20260810 04:16:41.976552 25861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.976548 25863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:41.976548 25860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:41.977141 25854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:41.977470 25854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:41.978657 25854 hybrid_clock.cc:648] HybridClock initialized: now 1786335401978640 us; error 52 us; skew 500 ppm
Aug 10 04:16:41 dist-test-slave-g350 krb5kdc[24937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335401, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:41.981849 25854 init.cc:377] Logged in from keytab as oryx/127.15.216.67@KRBTEST.COM (short username oryx)
I20260810 04:16:41.982362 25854 webserver.cc:460] Webserver started at http://127.15.216.67:44685/ using document root <none> and password file <none>
I20260810 04:16:41.982635 25854 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:41.982729 25854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:41.984411 25854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:16:41.985185 25870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:41.985431 25854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:41.985546 25854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "09ad3ba64fc542689f1c1d10281f51bb"
format_stamp: "Formatted at 2026-08-10 04:16:41 on dist-test-slave-g350"
server_key: "94093ca3cda3b5bb3e61fe1eb1aa44ab"
server_key_iv: "17fc232a1f0f83c278a0241cfbc49285"
server_key_version: "encryptionkey@0"
I20260810 04:16:41.986002 25854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:41.998461 25854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:42.001053 25854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:42.001263 25854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:42.001696 25854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:42.002081 25854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:42.002143 25854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:42.002182 25854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:42.002213 25854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:42.009476 25854 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:35175
I20260810 04:16:42.009565 25993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:35175 every 8 connection(s)
I20260810 04:16:42.010042 25854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:42.010581 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 25854
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 04:16:42.019500 25715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:42.011016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:39661 (local address 127.15.216.126:39089)
0810 04:16:42.011081 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:42.011084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:42.011348 (+ 264us) server_negotiation.cc:407] Connection header received
0810 04:16:42.012192 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:42.012193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:42.012202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:42.012241 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:42.013487 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.014091 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.015555 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.015757 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.015821 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.019432 (+ 3611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:39661: BlockingRecv error: recv got EOF from 127.15.216.67:39661 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:16:42.019569 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:42.011014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:39661)
0810 04:16:42.011194 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:42.011241 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:42.012069 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:42.012336 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:42.012341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:42.013033 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:42.013365 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.013372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.014207 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.014208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.015448 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.015451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.015870 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.015871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.016049 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.016802 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:42.016809 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:42.019331 (+ 2522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260810 04:16:42.019829 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:42.011199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.15.216.126:39089)
0810 04:16:42.011245 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:42.011247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:42.011432 (+ 185us) server_negotiation.cc:407] Connection header received
0810 04:16:42.011504 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:42.011505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:42.011518 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:42.011568 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:42.012465 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.013010 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.014118 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.014461 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.014533 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.016403 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:42.016419 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:42.016419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:42.016437 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:42.018358 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:42.018784 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:42.018787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:42.018787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:42.018879 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:42.019112 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:42.019112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:42.019112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:42.019342 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:42.019485 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:42.019618 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 04:16:42.019754 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 04:16:42.019879 25994 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:39089 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.216.126:39089: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:42.024528 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:42.020135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:33787)
0810 04:16:42.020169 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:42.020179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:42.020217 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:42.020365 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:42.020366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:42.020516 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:42.020574 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.020576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.021347 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.021349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.022021 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.022023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.022393 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.022394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.022452 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.022870 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:42.022874 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:42.024478 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 04:16:42.024662 25713 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:42.020158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:33787 (local address 127.15.216.126:39089)
0810 04:16:42.020191 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:42.020192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:42.020203 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:42.020278 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:42.020279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:42.020288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:42.020339 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:42.020653 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.021210 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.022099 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.022299 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.022358 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.024585 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:33787: BlockingRecv error: recv got EOF from 127.15.216.67:33787 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:42.029445 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:42.024856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:60987)
0810 04:16:42.024936 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:42.024950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:42.025004 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:42.025148 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:42.025149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:42.025340 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:42.025370 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.025372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.026082 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.026083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.026578 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.026580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.026953 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.026954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.027016 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.027476 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:42.027480 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:42.029349 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 04:16:42.029500 25713 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:42.024891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:60987 (local address 127.15.216.126:39089)
0810 04:16:42.024947 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:42.024950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:42.024988 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:16:42.025063 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:42.025064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:42.025072 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:42.025144 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:42.025453 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.025998 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.026667 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.026876 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.026954 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.029437 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:60987: BlockingRecv error: recv got EOF from 127.15.216.67:60987 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 04:16:42.029688 25994 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:42.754851 25998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:42.749840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:57047)
0810 04:16:42.749991 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:42.750002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:42.750051 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:42.750227 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:42.750228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:42.750410 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:42.750458 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.750460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.751161 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.751162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.751733 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.751735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.752106 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.752106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.752149 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.752616 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:42.752621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:42.754777 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260810 04:16:42.755102 25999 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:42.749911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:57047 (local address 127.15.216.126:39089)
0810 04:16:42.750052 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:42.750054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:42.750112 (+ 58us) server_negotiation.cc:407] Connection header received
0810 04:16:42.750140 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:42.750141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:42.750151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:42.750205 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:42.750537 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.751058 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.751823 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.752031 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.752089 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.755023 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:57047: BlockingRecv error: recv got EOF from 127.15.216.65:57047 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:42 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:42.893960 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:42.888630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:56421)
0810 04:16:42.888871 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:42.888881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:42.888943 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:42.889082 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:42.889083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:42.889301 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:42.889367 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.889369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.890148 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.890149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.890755 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:42.890758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.891126 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.891128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:42.891194 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.891725 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:42.891730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:42.893847 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":175,"threads_started":1}
W20260810 04:16:42.893987 25999 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:42.888725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56421 (local address 127.15.216.126:39089)
0810 04:16:42.888793 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:42.888795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:42.888942 (+ 147us) server_negotiation.cc:407] Connection header received
0810 04:16:42.888990 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:42.888991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:42.889003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:42.889067 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:42.889447 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.890068 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.890828 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:42.891015 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:42.891115 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:42.893922 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:56421: BlockingRecv error: recv got EOF from 127.15.216.66:56421 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 04:16:43 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:43 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:43.035388 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:43.030156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:44159)
0810 04:16:43.030373 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:43.030386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:43.030467 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:43.030623 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:43.030624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:43.030823 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:43.030895 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:43.030896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.031586 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.031587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:43.032149 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:43.032151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.032522 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.032523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:43.032582 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:43.033102 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:43.033115 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:43.035259 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":122,"threads_started":1}
W20260810 04:16:43.035576 25999 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:43.030284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:44159 (local address 127.15.216.126:39089)
0810 04:16:43.030401 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:43.030403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:43.030444 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:16:43.030523 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:43.030525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:43.030538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:43.030614 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:43.030987 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.031494 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.032243 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.032445 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.032523 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:43.035489 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:44159: BlockingRecv error: recv got EOF from 127.15.216.67:44159 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 04:16:43 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:43 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:43.760545 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:43.755436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:36551)
0810 04:16:43.755607 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:43.755618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:43.755677 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:43.755900 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:43.755901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:43.756093 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:43.756153 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:43.756155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.756876 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.756877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:43.757453 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:43.757454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.757937 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.757938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:43.757994 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:43.758536 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:43.758540 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:43.760442 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
W20260810 04:16:43.760612 26005 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:43.755535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:36551 (local address 127.15.216.126:39089)
0810 04:16:43.755681 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:43.755683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:43.755693 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:43.755747 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:43.755748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:43.755759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:43.755824 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:43.756247 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.756788 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.757553 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.757835 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.757906 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:43.760531 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:36551: BlockingRecv error: recv got EOF from 127.15.216.65:36551 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
Aug 10 04:16:43 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:43 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:43.900166 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:43.894503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:45893)
0810 04:16:43.894806 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:43.894819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:43.894925 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:43.895102 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:43.895103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:43.895388 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:43.895440 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:43.895443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.896278 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.896279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:43.897001 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:43.897005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.897410 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.897411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:43.897467 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:43.898049 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:43.898055 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:43.900104 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":93,"threads_started":1}
W20260810 04:16:43.900341 26005 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:43.894608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:45893 (local address 127.15.216.126:39089)
0810 04:16:43.894800 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:43.894803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:43.894904 (+ 101us) server_negotiation.cc:407] Connection header received
0810 04:16:43.894992 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:43.894993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:43.895010 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:43.895096 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:43.895547 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.896121 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.897118 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:43.897331 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:43.897411 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:43.900267 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:45893: BlockingRecv error: recv got EOF from 127.15.216.66:45893 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Aug 10 04:16:44 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:44 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:44.041159 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:44.036002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:38645)
0810 04:16:44.036156 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:44.036188 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:44.036267 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:44.036486 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:44.036487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:44.036682 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:44.036735 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:44.036737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.037485 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.037486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:44.038049 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:44.038051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.038473 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.038474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:44.038536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:44.039157 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:44.039162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:44.041071 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260810 04:16:44.041352 26005 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:44.036138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:38645 (local address 127.15.216.126:39089)
0810 04:16:44.036218 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:44.036220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:44.036246 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:16:44.036382 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:44.036383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:44.036405 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:44.036478 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:44.036831 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.037388 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.038145 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.038391 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.038473 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:44.041207 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:38645: BlockingRecv error: recv got EOF from 127.15.216.67:38645 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 04:16:44 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:44 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:44.766182 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:44.761099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:60021)
0810 04:16:44.761252 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:44.761261 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:44.761327 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:44.761606 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:44.761607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:44.761775 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:44.761830 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:44.761832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.762611 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.762612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:44.763152 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:44.763154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.763598 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.763599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:44.763652 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:44.764255 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:44.764262 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:44.766123 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
W20260810 04:16:44.766371 26009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:44.761307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:60021 (local address 127.15.216.126:39089)
0810 04:16:44.761442 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:44.761447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:44.761462 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:16:44.761486 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:44.761486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:44.761498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:44.761571 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:44.761932 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.762527 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.763252 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.763505 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.763606 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:44.766277 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:60021: BlockingRecv error: recv got EOF from 127.15.216.65:60021 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
Aug 10 04:16:44 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:44 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:44.905372 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:44.900667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:56707)
0810 04:16:44.900803 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:44.900819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:44.900866 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:44.901049 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:44.901050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:44.901216 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:44.901266 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:44.901268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.902023 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.902024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:44.902553 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:44.902555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.902915 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.902915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:44.902955 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:44.903421 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:44.903425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:44.905274 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
W20260810 04:16:44.905421 26009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:44.900786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56707 (local address 127.15.216.126:39089)
0810 04:16:44.900850 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:44.900852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:44.900862 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:44.900963 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:44.900965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:44.900977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:44.901035 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:44.901388 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.901932 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.902643 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:44.902843 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:44.902905 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:44.905359 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:56707: BlockingRecv error: recv got EOF from 127.15.216.66:56707 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 04:16:45 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:45 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:45.046456 26011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:45.041793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:46135)
0810 04:16:45.041947 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:45.041959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:45.042027 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:45.042169 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:45.042170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:45.042398 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:45.042453 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:45.042455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.043084 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.043085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:45.043646 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:45.043648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.043971 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.043972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:45.044016 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:45.044475 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:45.044480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:45.046358 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260810 04:16:45.046536 26009 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:45.041908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:46135 (local address 127.15.216.126:39089)
0810 04:16:45.041960 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:45.041962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:45.042012 (+ 50us) server_negotiation.cc:407] Connection header received
0810 04:16:45.042070 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:45.042071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:45.042084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:45.042149 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:45.042519 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.043006 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.043716 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.043901 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.043954 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:45.046462 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:46135: BlockingRecv error: recv got EOF from 127.15.216.67:46135 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 10 04:16:45 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:45 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:45.774695 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:45.766679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:37025)
0810 04:16:45.766859 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:45.766870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:45.766923 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:45.767064 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:45.767064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:45.767288 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:45.767341 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:45.767343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.768139 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.768140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:45.768690 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:45.768692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.769095 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.769097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:45.769142 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:45.772609 (+ 3467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:45.772614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:45.774579 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
W20260810 04:16:45.774772 26013 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:45.766784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:37025 (local address 127.15.216.126:39089)
0810 04:16:45.766947 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:45.766949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:45.766957 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:45.766978 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:45.766978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:45.766989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:45.767046 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:45.767501 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.768039 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.768799 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.769012 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.769084 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:45.774692 (+ 5608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:37025: BlockingRecv error: recv got EOF from 127.15.216.65:37025 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
Aug 10 04:16:45 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:45 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:45.910692 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:45.905929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:55987)
0810 04:16:45.906104 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:45.906122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:45.906186 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:45.906307 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:45.906308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:45.906487 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:45.906541 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:45.906543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.907201 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.907203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:45.907792 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:45.907795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.908144 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.908146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:45.908191 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:45.908693 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:45.908698 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:45.910594 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
W20260810 04:16:45.910734 26013 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:45.906024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:55987 (local address 127.15.216.126:39089)
0810 04:16:45.906083 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:45.906085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:45.906184 (+ 99us) server_negotiation.cc:407] Connection header received
0810 04:16:45.906236 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:45.906236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:45.906247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:45.906303 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:45.906606 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.907102 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.907878 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:45.908068 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:45.908130 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:45.910672 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:55987: BlockingRecv error: recv got EOF from 127.15.216.66:55987 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 04:16:46 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:46 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:46.051816 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:46.047078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:51509)
0810 04:16:46.047269 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:46.047280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:46.047385 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:46.047542 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:46.047543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:46.047721 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:46.047784 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:46.047786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.048409 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.048410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:46.048928 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:46.048930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.049273 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.049274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:46.049316 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:46.049793 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:46.049797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:46.051717 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":103,"threads_started":1}
W20260810 04:16:46.051877 26013 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:46.047135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:51509 (local address 127.15.216.126:39089)
0810 04:16:46.047209 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:46.047211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:46.047364 (+ 153us) server_negotiation.cc:407] Connection header received
0810 04:16:46.047448 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:46.047449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:46.047461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:46.047516 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:46.047846 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.048334 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.049015 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.049202 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.049259 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:46.051811 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:51509: BlockingRecv error: recv got EOF from 127.15.216.67:51509 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 04:16:46 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:46 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:46.781078 26016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:46.775336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:39629)
0810 04:16:46.775547 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:46.775562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:46.775734 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:46.776024 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:46.776025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:46.776209 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:46.776260 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:46.776262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.776995 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.776995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:46.777559 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:46.777561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.778114 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.778115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:46.778168 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:46.778723 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:46.778728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:46.781013 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
W20260810 04:16:46.781286 26017 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:46.775496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:39629 (local address 127.15.216.126:39089)
0810 04:16:46.775649 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:46.775651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:46.775662 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:46.775864 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:46.775866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:46.775884 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:46.775975 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:46.776374 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.776912 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.777750 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.778008 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.778083 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:46.781189 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:39629: BlockingRecv error: recv got EOF from 127.15.216.65:39629 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
Aug 10 04:16:46 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:46 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:46.916064 26018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:46.911266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:58161)
0810 04:16:46.911473 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:46.911485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:46.911550 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:46.911711 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:46.911713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:46.911884 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:46.911937 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:46.911938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.912589 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.912590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:46.913107 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:46.913109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.913619 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.913620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:46.913698 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:46.914207 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:46.914213 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:46.916002 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":111,"threads_started":1}
W20260810 04:16:46.916216 26017 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:46.911431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:58161 (local address 127.15.216.126:39089)
0810 04:16:46.911501 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:46.911502 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:46.911538 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:16:46.911619 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:46.911620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:46.911632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:46.911685 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:46.912002 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.912512 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.913183 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:46.913499 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:46.913574 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:46.916132 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:58161: BlockingRecv error: recv got EOF from 127.15.216.66:58161 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 04:16:47 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:47 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:47.057662 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:47.052400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:44977)
0810 04:16:47.052610 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:47.052621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:47.052702 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:47.052893 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:47.052894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:47.053091 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:47.053136 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:47.053138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.053922 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.053923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:47.054501 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:47.054503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.054858 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.054859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:47.054926 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:47.055450 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:47.055456 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:47.057582 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":108,"threads_started":1}
W20260810 04:16:47.057890 26017 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:47.052497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:44977 (local address 127.15.216.126:39089)
0810 04:16:47.052579 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:47.052582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:47.052681 (+ 99us) server_negotiation.cc:407] Connection header received
0810 04:16:47.052792 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:47.052793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:47.052808 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:47.052882 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:47.053225 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.053832 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.054588 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.054781 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.054858 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:47.057845 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:44977: BlockingRecv error: recv got EOF from 127.15.216.67:44977 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 04:16:47 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:47 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:47.787146 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:47.781801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:60647)
0810 04:16:47.781989 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:47.782003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:47.782087 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:47.782244 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:47.782244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:47.782423 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:47.782475 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:47.782478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.783208 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.783209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:47.783922 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:47.783925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.784335 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.784337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:47.784413 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:47.784965 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:47.785023 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:47.787083 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
W20260810 04:16:47.787268 26021 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:47.781880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:60647 (local address 127.15.216.126:39089)
0810 04:16:47.782068 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:47.782070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:47.782081 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:47.782138 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:47.782139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:47.782150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:47.782220 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:47.782570 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.783133 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.784002 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.784236 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.784342 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:47.787179 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:60647: BlockingRecv error: recv got EOF from 127.15.216.65:60647 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
Aug 10 04:16:47 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:47 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:47.921694 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:47.916664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:54631)
0810 04:16:47.916904 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:47.916924 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:47.916995 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:47.917144 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:47.917145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:47.917330 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:47.917383 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:47.917385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.918093 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.918093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:47.918627 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:47.918629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.919021 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.919022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:47.919088 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:47.919569 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:47.919574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:47.921560 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":184,"threads_started":1}
W20260810 04:16:47.921860 26021 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:47.916749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:54631 (local address 127.15.216.126:39089)
0810 04:16:47.916875 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:47.916877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:47.916990 (+ 113us) server_negotiation.cc:407] Connection header received
0810 04:16:47.917051 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:47.917052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:47.917069 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:47.917124 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:47.917452 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.918006 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.918730 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:47.918939 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:47.919020 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:47.921774 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:54631: BlockingRecv error: recv got EOF from 127.15.216.66:54631 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 10 04:16:48 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:48 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:48.063086 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:48.058267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:47663)
0810 04:16:48.058458 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:48.058470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:48.058536 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:48.058695 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:48.058696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:48.058881 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:48.058935 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:48.058937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.059608 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.059609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:48.060150 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:48.060152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.060575 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.060577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:48.060650 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:48.061203 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:48.061209 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:48.063023 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":116,"threads_started":1}
W20260810 04:16:48.063186 26021 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:48.058377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:47663 (local address 127.15.216.126:39089)
0810 04:16:48.058448 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:48.058455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:48.058534 (+ 79us) server_negotiation.cc:407] Connection header received
0810 04:16:48.058584 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:48.058585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:48.058597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:48.058654 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:48.059006 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.059511 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.060228 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.060456 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.060529 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:48.063118 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:47663: BlockingRecv error: recv got EOF from 127.15.216.67:47663 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 04:16:48 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:48 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:48.793279 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:48.787745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:58017)
0810 04:16:48.787914 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:48.787925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:48.787980 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:48.788156 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:48.788157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:48.788348 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:48.788401 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:48.788403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.789072 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.789073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:48.789807 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:48.789810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.790374 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.790375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:48.790445 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:48.791045 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:48.791052 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:48.793204 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
W20260810 04:16:48.793468 26025 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:48.787844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:58017 (local address 127.15.216.126:39089)
0810 04:16:48.787958 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:48.787960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:48.787969 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:16:48.788042 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:48.788042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:48.788054 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:48.788105 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:48.788494 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.788994 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.789908 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.790238 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.790308 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:48.793347 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:58017: BlockingRecv error: recv got EOF from 127.15.216.65:58017 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
Aug 10 04:16:48 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:48 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:48.927172 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:48.922243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:60723)
0810 04:16:48.922486 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:48.922497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:48.922563 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:48.922723 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:48.922724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:48.922920 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:48.922976 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:48.922978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.923678 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.923679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:48.924226 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:48.924227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.924591 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.924592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:48.924640 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:48.925139 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:48.925144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:48.927101 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":168,"threads_started":1}
W20260810 04:16:48.927304 26025 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:48.922319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:60723 (local address 127.15.216.126:39089)
0810 04:16:48.922392 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:48.922394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:48.922561 (+ 167us) server_negotiation.cc:407] Connection header received
0810 04:16:48.922627 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:48.922628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:48.922639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:48.922690 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:48.923076 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.923585 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.924308 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:48.924501 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:48.924553 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:48.927216 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:60723: BlockingRecv error: recv got EOF from 127.15.216.66:60723 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 04:16:49 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:49 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:49.069024 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:49.063641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:50677)
0810 04:16:49.063837 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:49.063849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:49.063922 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:49.064114 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:49.064115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:49.064365 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:49.064418 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:49.064420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.065141 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.065142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:49.065895 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:49.065899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.066293 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.066294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:49.066351 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:49.066834 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:49.066839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:49.068940 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":130,"threads_started":1}
W20260810 04:16:49.069169 26025 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:49.063712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:50677 (local address 127.15.216.126:39089)
0810 04:16:49.063794 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:49.063796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:49.063972 (+ 176us) server_negotiation.cc:407] Connection header received
0810 04:16:49.064003 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:49.064004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:49.064020 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:49.064092 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:49.064502 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.065035 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.065995 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.066185 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.066250 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:49.069053 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:50677: BlockingRecv error: recv got EOF from 127.15.216.67:50677 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 04:16:49 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:49 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:49.799072 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:49.793878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:56427)
0810 04:16:49.794045 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:49.794057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:49.794116 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:49.794374 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:49.794375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:49.794546 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:49.794597 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:49.794598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.795640 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.795642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:49.796196 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:49.796198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.796611 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.796612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:49.796659 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:49.797148 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:49.797153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:49.799010 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
W20260810 04:16:49.799260 26030 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:49.794017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:56427 (local address 127.15.216.126:39089)
0810 04:16:49.794166 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:49.794168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:49.794176 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:16:49.794198 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:49.794198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:49.794210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:49.794337 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:49.794715 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.795559 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.796311 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.796535 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.796596 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:49.799149 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:56427: BlockingRecv error: recv got EOF from 127.15.216.65:56427 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
Aug 10 04:16:49 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:49 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:49.933143 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:49.927824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:47311)
0810 04:16:49.928009 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:49.928021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:49.928123 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:49.928316 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:49.928317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:49.928516 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:49.928569 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:49.928571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.929467 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.929469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:49.930064 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:49.930067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.930493 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.930494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:49.930572 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:49.931092 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:49.931097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:49.933082 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":103,"threads_started":1}
W20260810 04:16:49.933280 26030 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:49.927924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:47311 (local address 127.15.216.126:39089)
0810 04:16:49.928007 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:49.928009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:49.928100 (+ 91us) server_negotiation.cc:407] Connection header received
0810 04:16:49.928225 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:49.928227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:49.928242 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:49.928296 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:49.928672 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.929363 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.930175 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:49.930396 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:49.930491 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:49.933190 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:47311: BlockingRecv error: recv got EOF from 127.15.216.66:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 04:16:50 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:50 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:50.074508 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:50.069640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:57901)
0810 04:16:50.069809 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:50.069827 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:50.069910 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:50.070046 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:50.070046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:50.070214 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:50.070265 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:50.070266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.070910 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.070912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:50.071444 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:50.071446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.071875 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.071876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:50.071925 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:50.072428 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:50.072433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:50.074444 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
W20260810 04:16:50.074731 26030 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:50.069738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:57901 (local address 127.15.216.126:39089)
0810 04:16:50.069821 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:50.069823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:50.069937 (+ 114us) server_negotiation.cc:407] Connection header received
0810 04:16:50.069965 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:50.069965 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:50.069979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:50.070041 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:50.070330 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.070826 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.071561 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.071780 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.071852 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:50.074647 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:57901: BlockingRecv error: recv got EOF from 127.15.216.67:57901 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 04:16:50 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:50 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:50.812330 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:50.799603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:54351)
0810 04:16:50.799854 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:50.799867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:50.799939 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:50.800122 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:50.800123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:50.800318 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:50.800368 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:50.800370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.801074 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.801075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:50.801678 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:50.801680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.802053 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.802054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:50.802099 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:50.809646 (+ 7547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:50.809656 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:50.812230 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":114,"threads_started":1}
W20260810 04:16:50.812566 26034 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:50.799726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:54351 (local address 127.15.216.126:39089)
0810 04:16:50.799857 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:50.799859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:50.799925 (+ 66us) server_negotiation.cc:407] Connection header received
0810 04:16:50.799989 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:50.799990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:50.800002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:50.800069 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:50.800479 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.800993 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.801775 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.801977 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.802037 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:50.812461 (+ 10424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:54351: BlockingRecv error: recv got EOF from 127.15.216.65:54351 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
Aug 10 04:16:50 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:50 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:50.938710 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:50.933799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:53745)
0810 04:16:50.933933 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:50.933943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:50.934006 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:50.934192 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:50.934193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:50.934467 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:50.934552 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:50.934556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.935300 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.935300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:50.935830 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:50.935832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.936250 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.936251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:50.936304 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:50.936738 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:50.936743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:50.938642 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260810 04:16:50.938855 26034 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:50.933839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:53745 (local address 127.15.216.126:39089)
0810 04:16:50.933915 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:50.933918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:50.934003 (+ 85us) server_negotiation.cc:407] Connection header received
0810 04:16:50.934083 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:50.934085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:50.934100 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:50.934168 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:50.934636 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.935206 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.935922 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:50.936117 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:50.936175 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:50.938736 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:53745: BlockingRecv error: recv got EOF from 127.15.216.66:53745 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 04:16:51 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:51 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:51.080443 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:51.075072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:34271)
0810 04:16:51.075240 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:51.075252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:51.075353 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:51.075590 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:51.075591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:51.075846 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:51.075943 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:51.075945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.076780 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.076781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:51.077341 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:51.077343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.077761 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.077762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:51.077810 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:51.078368 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:51.078373 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:51.080371 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
W20260810 04:16:51.080576 26034 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:51.075207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:34271 (local address 127.15.216.126:39089)
0810 04:16:51.075289 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:51.075291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:51.075301 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:51.075459 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:51.075460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:51.075477 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:51.075558 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:51.076055 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.076694 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.077439 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.077658 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.077742 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:51.080486 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:34271: BlockingRecv error: recv got EOF from 127.15.216.67:34271 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 04:16:51 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:51 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:51.817970 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:51.812972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:38201)
0810 04:16:51.813231 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:51.813245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:51.813325 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:51.813511 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:51.813512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:51.813717 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:51.813770 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:51.813772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.814472 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.814473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:51.815017 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:51.815018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.815420 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.815421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:51.815476 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:51.816007 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:51.816012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:51.817897 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
W20260810 04:16:51.818112 26038 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:51.813082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:38201 (local address 127.15.216.126:39089)
0810 04:16:51.813211 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:51.813214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:51.813330 (+ 116us) server_negotiation.cc:407] Connection header received
0810 04:16:51.813388 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:51.813389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:51.813400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:51.813460 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:51.813846 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.814382 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.815103 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.815326 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.815384 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:51.818000 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:38201: BlockingRecv error: recv got EOF from 127.15.216.65:38201 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
Aug 10 04:16:51 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:51 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:51.946358 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:51.939250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:58711)
0810 04:16:51.939441 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:51.939456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:51.939519 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:51.939763 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:51.939764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:51.940008 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:51.940078 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:51.940081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.941478 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.941480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:51.942515 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:51.942519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.943196 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.943197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:51.943260 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:51.943856 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:51.943862 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:51.946253 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
W20260810 04:16:51.946590 26038 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:51.939401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:58711 (local address 127.15.216.126:39089)
0810 04:16:51.939487 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:51.939489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:51.939568 (+ 79us) server_negotiation.cc:407] Connection header received
0810 04:16:51.939634 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:51.939654 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:51.939670 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:51.939747 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:51.940265 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.941375 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.942640 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:51.943092 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:51.943196 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:51.946452 (+ 3256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:58711: BlockingRecv error: recv got EOF from 127.15.216.66:58711 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 04:16:52 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:52 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:52.086658 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:52.081106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:35931)
0810 04:16:52.081289 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:52.081354 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:52.081433 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:52.081653 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:52.081654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:52.081857 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:52.081908 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:52.081910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.082609 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.082611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:52.083421 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:52.083423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.083797 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.083798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:52.083861 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:52.084329 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:52.084334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:52.086540 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
W20260810 04:16:52.086705 26038 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:52.081281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:35931 (local address 127.15.216.126:39089)
0810 04:16:52.081394 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:52.081396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:52.081410 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:16:52.081520 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:52.081521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:52.081537 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:52.081650 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:52.082005 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.082505 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.083513 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.083707 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.083793 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:52.086638 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:35931: BlockingRecv error: recv got EOF from 127.15.216.67:35931 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 04:16:52 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:52 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:52.823597 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:52.818635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:44713)
0810 04:16:52.818834 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:52.818848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:52.818906 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:52.819117 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:52.819118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:52.819429 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:52.819480 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:52.819482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.820155 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.820155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:52.820698 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:52.820699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.821054 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.821055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:52.821109 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:52.821569 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:52.821574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:52.823521 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":133,"threads_started":1}
W20260810 04:16:52.823669 26042 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:52.818736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:44713 (local address 127.15.216.126:39089)
0810 04:16:52.818877 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:52.818879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:52.818889 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:52.818987 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:52.818988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:52.819001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:52.819103 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:52.819570 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.820083 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.820775 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.820964 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.821016 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:52.823616 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:44713: BlockingRecv error: recv got EOF from 127.15.216.65:44713 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
Aug 10 04:16:52 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:52 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:52.951982 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:52.946961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:48931)
0810 04:16:52.947132 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:52.947143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:52.947209 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:52.947386 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:52.947387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:52.947655 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:52.947712 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:52.947714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.948413 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.948414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:52.948959 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:52.948961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.949343 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.949344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:52.949383 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:52.949985 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:52.949992 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:52.951873 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
W20260810 04:16:52.952116 26042 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:52.947027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:48931 (local address 127.15.216.126:39089)
0810 04:16:52.947127 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:52.947129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:52.947204 (+ 75us) server_negotiation.cc:407] Connection header received
0810 04:16:52.947292 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:52.947293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:52.947306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:52.947367 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:52.947784 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.948329 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.949035 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:52.949272 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:52.949332 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:52.952044 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:48931: BlockingRecv error: recv got EOF from 127.15.216.66:48931 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 04:16:53 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:53 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:53.092306 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:53.087183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:39319)
0810 04:16:53.087369 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:53.087382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:53.087461 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:53.087636 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:53.087637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:53.087842 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:53.087892 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:53.087893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.088683 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.088684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:53.089258 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:53.089260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.089736 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.089737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:53.089803 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:53.090348 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:53.090354 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:53.092231 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
W20260810 04:16:53.092417 26042 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:53.087236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:39319 (local address 127.15.216.126:39089)
0810 04:16:53.087373 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:53.087375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:53.087473 (+ 98us) server_negotiation.cc:407] Connection header received
0810 04:16:53.087543 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:53.087544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:53.087556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:53.087612 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:53.087982 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.088570 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.089364 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.089665 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.089722 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:53.092319 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:39319: BlockingRecv error: recv got EOF from 127.15.216.67:39319 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 10 04:16:53 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:53 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:53.830071 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:53.824242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:35797)
0810 04:16:53.824506 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:53.824526 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:53.824583 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:53.824965 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:53.824967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:53.825165 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:53.825213 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:53.825215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.825996 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.825997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:53.826613 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:53.826615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.826997 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.826998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:53.827052 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:53.827683 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:53.827688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:53.829998 (+ 2310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":197,"threads_started":1}
W20260810 04:16:53.830207 26046 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:53.824465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:35797 (local address 127.15.216.126:39089)
0810 04:16:53.824627 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:53.824629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:53.824639 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:16:53.824676 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:53.824677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:53.824689 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:53.824771 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:53.825328 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.825901 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.826716 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.826916 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.826975 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:53.830113 (+ 3138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:35797: BlockingRecv error: recv got EOF from 127.15.216.65:35797 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
Aug 10 04:16:53 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:53 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:53.957847 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:53.952465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:56145)
0810 04:16:53.952642 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:53.952657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:53.952765 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:53.952896 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:53.952897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:53.953092 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:53.953139 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:53.953141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.953959 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.953960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:53.954537 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:53.954539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.955030 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.955031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:53.955085 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:53.955637 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:53.955642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:53.957763 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260810 04:16:53.957922 26046 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:53.952584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56145 (local address 127.15.216.126:39089)
0810 04:16:53.952667 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:53.952670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:53.952741 (+ 71us) server_negotiation.cc:407] Connection header received
0810 04:16:53.952826 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:53.952827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:53.952839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:53.952891 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:53.953252 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.953854 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.954688 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:53.954912 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:53.954990 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:53.957869 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:56145: BlockingRecv error: recv got EOF from 127.15.216.66:56145 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 04:16:54 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:54 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:54.097391 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:54.092765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:55639)
0810 04:16:54.092919 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:54.092936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:54.092993 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:54.093148 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:54.093149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:54.093330 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:54.093381 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:54.093383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.094031 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.094032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:54.094554 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:54.094555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.094896 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.094897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:54.094962 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:54.095461 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:54.095467 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:54.097332 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260810 04:16:54.097496 26046 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:54.092877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:55639 (local address 127.15.216.126:39089)
0810 04:16:54.092950 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:54.092952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:54.092982 (+ 30us) server_negotiation.cc:407] Connection header received
0810 04:16:54.093050 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:54.093051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:54.093064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:54.093127 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:54.093449 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.093964 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.094623 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.094820 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.094873 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:54.097428 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:55639: BlockingRecv error: recv got EOF from 127.15.216.67:55639 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 04:16:54 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:54 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:54.835832 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:54.830623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:57461)
0810 04:16:54.830818 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:54.830830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:54.830896 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:54.831038 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:54.831038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:54.831306 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:54.831359 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:54.831362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.832057 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.832058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:54.832664 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:54.832666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.833033 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.833034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:54.833077 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:54.833706 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:54.833713 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:54.835768 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":117,"threads_started":1}
W20260810 04:16:54.835973 26051 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:54.830768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:57461 (local address 127.15.216.126:39089)
0810 04:16:54.830904 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:54.830906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:54.830917 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:54.830941 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:54.830942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:54.830952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:54.831009 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:54.831446 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.831964 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.832751 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.832963 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.833016 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:54.835861 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:57461: BlockingRecv error: recv got EOF from 127.15.216.65:57461 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
Aug 10 04:16:54 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:54 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:54.963975 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:54.958495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:46419)
0810 04:16:54.958700 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:54.958710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:54.958768 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:54.958931 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:54.958931 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:54.959108 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:54.959160 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:54.959162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.959897 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.959898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:54.960717 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:54.960720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.961185 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.961186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:54.961263 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:54.961917 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:54.961923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:54.963907 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":124,"threads_started":1}
W20260810 04:16:54.964138 26051 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:54.958660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:46419 (local address 127.15.216.126:39089)
0810 04:16:54.958735 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:54.958737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:54.958758 (+ 21us) server_negotiation.cc:407] Connection header received
0810 04:16:54.958834 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:54.958834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:54.958845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:54.958896 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:54.959299 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.959802 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.960844 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:54.961108 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:54.961162 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:54.964045 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:46419: BlockingRecv error: recv got EOF from 127.15.216.66:46419 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 04:16:55 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:55 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:55.102985 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:55.097934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:60013)
0810 04:16:55.098149 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:55.098162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:55.098255 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:55.098496 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:55.098497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:55.098700 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:55.098748 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:55.098750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.099449 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.099450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:55.100015 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:55.100017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.100384 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.100385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:55.100446 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:55.100978 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:55.100983 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:55.102874 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":127,"threads_started":1}
W20260810 04:16:55.103024 26051 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:55.098050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:60013 (local address 127.15.216.126:39089)
0810 04:16:55.098133 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:55.098136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:55.098240 (+ 104us) server_negotiation.cc:407] Connection header received
0810 04:16:55.098398 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:55.098399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:55.098417 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:55.098486 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:55.098841 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.099355 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.100106 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.100306 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.100385 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:55.102956 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:60013: BlockingRecv error: recv got EOF from 127.15.216.67:60013 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 04:16:55 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:55 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:55.847129 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:55.836505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:50281)
0810 04:16:55.836638 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:55.836650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:55.836713 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:55.836935 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:55.836936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:55.837119 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:55.837171 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:55.837172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.838001 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.838002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:55.838617 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:55.838620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.839085 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.839086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:55.839160 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:55.844820 (+ 5660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:55.844825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:55.847024 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260810 04:16:55.847172 26055 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:55.836598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:50281 (local address 127.15.216.126:39089)
0810 04:16:55.836772 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:55.836775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:55.836786 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:55.836827 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:55.836827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:55.836839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:55.836916 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:55.837257 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.837913 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.838735 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.838958 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.839070 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:55.847114 (+ 8044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:50281: BlockingRecv error: recv got EOF from 127.15.216.65:50281 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
Aug 10 04:16:55 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:55 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:55.969563 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:55.964494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:49601)
0810 04:16:55.964652 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:55.964664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:55.964749 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:55.964908 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:55.964910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:55.965154 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:55.965206 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:55.965209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.965900 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.965901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:55.966435 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:55.966436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.966805 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.966807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:55.966850 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:55.967526 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:55.967531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:55.969468 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260810 04:16:55.969715 26055 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:55.964583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:49601 (local address 127.15.216.126:39089)
0810 04:16:55.964656 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:55.964658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:55.964733 (+ 75us) server_negotiation.cc:407] Connection header received
0810 04:16:55.964798 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:55.964799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:55.964810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:55.964869 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:55.965286 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.965823 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.966519 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:55.966732 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:55.966790 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:55.969561 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:49601: BlockingRecv error: recv got EOF from 127.15.216.66:49601 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 04:16:56 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:56 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.67@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:56.108604 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:56.103601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.67:33335)
0810 04:16:56.103808 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:56.103821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:56.103887 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:56.104105 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:56.104106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:56.104304 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:56.104357 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:56.104358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.105066 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.105067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:56.105716 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:56.105718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.106115 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.106116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:56.106171 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:56.106630 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:56.106635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:56.108542 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":116,"threads_started":1}
W20260810 04:16:56.108681 26055 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:56.103704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:33335 (local address 127.15.216.126:39089)
0810 04:16:56.103755 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:56.103758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:56.103899 (+ 141us) server_negotiation.cc:407] Connection header received
0810 04:16:56.103995 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:56.103996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:56.104018 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:56.104085 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:56.104451 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.104970 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.105844 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.106044 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.106097 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:56.108633 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.67:33335: BlockingRecv error: recv got EOF from 127.15.216.67:33335 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 10 04:16:56 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:56 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.65@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:56.853322 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:56.847942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.65:33481)
0810 04:16:56.848129 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:56.848140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:56.848195 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:56.848517 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:56.848518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:56.848707 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:56.848758 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:56.848760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.849532 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.849534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:56.850112 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:56.850114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.850520 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.850521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:56.850571 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:56.851111 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:56.851116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:56.853237 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
W20260810 04:16:56.853451 26059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:56.848086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:33481 (local address 127.15.216.126:39089)
0810 04:16:56.848324 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:56.848326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:56.848337 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:16:56.848368 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:56.848368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:56.848379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:56.848462 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:56.848888 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.849442 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.850204 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.850429 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.850489 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:56.853374 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.65:33481: BlockingRecv error: recv got EOF from 127.15.216.65:33481 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":177,"threads_started":1}
Aug 10 04:16:56 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 04:16:56 dist-test-slave-g350 krb5kdc[24937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.216.66@KRBTEST.COM for oryx/127.15.216.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 04:16:56.975399 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:56.970299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39089 (local address 127.15.216.66:45973)
0810 04:16:56.970471 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:56.970482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:56.970546 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:56.970676 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:56.970677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:56.970844 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:56.970896 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:56.970897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.971676 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.971677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:56.972207 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:56.972209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.972667 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.972667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:56.972714 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:56.973240 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:56.973246 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:56.975297 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.216.126:39089: Server oryx/127.15.216.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
W20260810 04:16:56.975459 26059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:56.970377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:45973 (local address 127.15.216.126:39089)
0810 04:16:56.970452 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:56.970454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:56.970548 (+ 94us) server_negotiation.cc:407] Connection header received
0810 04:16:56.970596 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:56.970597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:56.970608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:56.970672 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:56.970997 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.971593 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.972320 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:56.972590 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:56.972652 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:56.975394 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.216.66:45973: BlockingRecv error: recv got EOF from 127.15.216.66:45973 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:16:57.017655 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 25554
I20260810 04:16:57.025722 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 25714
I20260810 04:16:57.033293 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 25854
I20260810 04:16:57.040781 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 25481
2026-08-10T04:16:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17700 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T04:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:57Z Disabled control of system clock
I20260810 04:16:57.073936 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:46589
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 04:16:57.159467 26064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:57.159687 26064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:57.159711 26064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:57.160852 26064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:57.160898 26064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:57.160913 26064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:57.160928 26064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:57.160941 26064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:57.162267 26064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:46589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:57.162475 26064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:57.162735 26064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:57.168192 26069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.168157 26070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.168155 26072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:57.168184 26064 server_base.cc:1034] running on GCE node
I20260810 04:16:57.168666 26064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:57.168916 26064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:57.170053 26064 hybrid_clock.cc:648] HybridClock initialized: now 1786335417170028 us; error 42 us; skew 500 ppm
I20260810 04:16:57.171295 26064 webserver.cc:460] Webserver started at http://127.15.216.126:38621/ using document root <none> and password file <none>
I20260810 04:16:57.171564 26064 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:57.171613 26064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:57.171761 26064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:57.173345 26064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "497e2b81bd0f424580fa4c18f5976ced"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "d6fc43c44232b1bb7d74f7e716b8b094"
server_key_iv: "b3bcd5121117b5b012e95f304ac92e71"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.173796 26064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "497e2b81bd0f424580fa4c18f5976ced"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "d6fc43c44232b1bb7d74f7e716b8b094"
server_key_iv: "b3bcd5121117b5b012e95f304ac92e71"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.175139 26064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:57.176054 26078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.176239 26064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:57.176321 26064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "497e2b81bd0f424580fa4c18f5976ced"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "d6fc43c44232b1bb7d74f7e716b8b094"
server_key_iv: "b3bcd5121117b5b012e95f304ac92e71"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.176378 26064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:57.201404 26064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:57.201794 26064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:57.201921 26064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:57.206013 26064 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:46589
I20260810 04:16:57.206067 26135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:46589 every 8 connection(s)
I20260810 04:16:57.206439 26064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:57.207211 26136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:57.209157 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26064
I20260810 04:16:57.209263 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:57.209476 16225 external_mini_cluster.cc:1428] Setting key fcd669ee68189b91575eddcd3c929abe
I20260810 04:16:57.210441 26136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced: Bootstrap starting.
I20260810 04:16:57.211180 26136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:57.211530 26136 log.cc:824] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:57.212347 26136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced: No bootstrap required, opened a new log
I20260810 04:16:57.214882 26136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497e2b81bd0f424580fa4c18f5976ced" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 46589 } }
I20260810 04:16:57.215014 26136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:57.215037 26136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497e2b81bd0f424580fa4c18f5976ced, State: Initialized, Role: FOLLOWER
I20260810 04:16:57.215189 26136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497e2b81bd0f424580fa4c18f5976ced" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 46589 } }
I20260810 04:16:57.215297 26136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:57.215344 26136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:57.215394 26136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:57.215664 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.210128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.15.216.126:46589)
0810 04:16:57.210320 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:57.210327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:57.210357 (+ 30us) server_negotiation.cc:407] Connection header received
0810 04:16:57.210791 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:57.210798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:57.210868 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:57.210981 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:57.211277 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.212355 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.213502 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.213994 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.214086 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:57.214824 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:57.214827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:57.214828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:57.214993 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:57.215056 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:57.215072 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:57.215163 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:57.215243 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 04:16:57.215512 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260810 04:16:57.216117 26136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497e2b81bd0f424580fa4c18f5976ced" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 46589 } }
I20260810 04:16:57.216226 26136 leader_election.cc:302] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 497e2b81bd0f424580fa4c18f5976ced; no voters:
I20260810 04:16:57.216404 26136 leader_election.cc:288] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:57.216457 26141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:57.216624 26141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [term 1 LEADER]: Becoming Leader. State: Replica: 497e2b81bd0f424580fa4c18f5976ced, State: Running, Role: LEADER
I20260810 04:16:57.216717 26136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:57.216758 26141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497e2b81bd0f424580fa4c18f5976ced" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 46589 } }
I20260810 04:16:57.217162 26142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "497e2b81bd0f424580fa4c18f5976ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497e2b81bd0f424580fa4c18f5976ced" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 46589 } } }
I20260810 04:16:57.217250 26142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:57.217427 26143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 497e2b81bd0f424580fa4c18f5976ced. Latest consensus state: current_term: 1 leader_uuid: "497e2b81bd0f424580fa4c18f5976ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "497e2b81bd0f424580fa4c18f5976ced" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 46589 } } }
I20260810 04:16:57.217494 26143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:57.217708 26150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:57.218255 26150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:57.219798 26150 catalog_manager.cc:1347] Generated new cluster ID: 4e0bf80e69404db59043fafaa159ca4a
I20260810 04:16:57.219857 26150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:57.234788 26150 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:57.235502 26150 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:57.239114 26150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 497e2b81bd0f424580fa4c18f5976ced: Generated new TSK 0
I20260810 04:16:57.239295 26150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:16:57.246305 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:46589
--builtin_ntp_servers=127.15.216.84:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 04:16:57.333498 26160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:57.333819 26160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:57.333863 26160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:57.335032 26160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:57.335103 26160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:57.335150 26160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:57.336598 26160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:57.336889 26160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:57.337225 26160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:57.342736 26165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.342783 26169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.342729 26166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:57.343079 26160 server_base.cc:1034] running on GCE node
I20260810 04:16:57.343297 26160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:57.343585 26160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:57.344764 26160 hybrid_clock.cc:648] HybridClock initialized: now 1786335417344738 us; error 43 us; skew 500 ppm
I20260810 04:16:57.346360 26160 webserver.cc:460] Webserver started at http://127.15.216.65:38979/ using document root <none> and password file <none>
I20260810 04:16:57.346643 26160 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:57.346695 26160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:57.346891 26160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:57.348090 26160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "cc7cb020812946e8903343c66f4da722"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "128f68b89a7e2c758dc5b92e255e678b"
server_key_iv: "17e67347f6959d2d637f7a4a38f75245"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.348490 26160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "cc7cb020812946e8903343c66f4da722"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "128f68b89a7e2c758dc5b92e255e678b"
server_key_iv: "17e67347f6959d2d637f7a4a38f75245"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.349968 26160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:57.350841 26175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.351035 26160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:57.351151 26160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "cc7cb020812946e8903343c66f4da722"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "128f68b89a7e2c758dc5b92e255e678b"
server_key_iv: "17e67347f6959d2d637f7a4a38f75245"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.351240 26160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:57.359310 26160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:57.359620 26160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:57.359786 26160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:57.360023 26160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:57.360389 26160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:57.360433 26160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.360458 26160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:57.360473 26160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.367412 26160 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:44929
I20260810 04:16:57.367460 26295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:44929 every 8 connection(s)
I20260810 04:16:57.367794 26160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:57.372588 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26160
I20260810 04:16:57.372682 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:57.372907 16225 external_mini_cluster.cc:1428] Setting key 38a54292b054065fa7ef93040f744da1
I20260810 04:16:57.373266 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.368493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:36157 (local address 127.15.216.126:46589)
0810 04:16:57.368563 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:57.368565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:57.368807 (+ 242us) server_negotiation.cc:407] Connection header received
0810 04:16:57.369343 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:57.369345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:57.369356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:57.369406 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:57.370128 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.370642 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.371625 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.371854 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.371936 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:57.372847 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:57.372849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:57.372849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:57.372855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:57.372871 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:57.372875 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:57.372934 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:57.373165 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 04:16:57.373189 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:16:57.373400 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.368475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:46589 (local address 127.15.216.65:36157)
0810 04:16:57.368688 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:57.368720 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:57.369238 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:57.369468 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:57.369477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:57.369486 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:57.370021 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:57.370027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.370750 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.370751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:57.371529 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:57.371532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.371948 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.371949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:57.372082 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:57.372646 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:57.372647 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:57.372688 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:57.372716 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:57.372952 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:57.372959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:57.372962 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:57.373071 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:16:57.373179 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260810 04:16:57.373994 26296 heartbeater.cc:347] Connected to a master server at 127.15.216.126:46589
I20260810 04:16:57.374084 26296 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:57.374203 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:46589
--builtin_ntp_servers=127.15.216.84:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 04:16:57.374262 26296 heartbeater.cc:511] Master 127.15.216.126:46589 requested a full tablet report, sending...
I20260810 04:16:57.374795 26097 ts_manager.cc:194] Registered new tserver with Master: cc7cb020812946e8903343c66f4da722 (127.15.216.65:44929)
I20260810 04:16:57.375801 26097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.65:36157
W20260810 04:16:57.462548 26299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:57.462759 26299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:57.462780 26299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:57.463869 26299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:57.463905 26299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:57.463929 26299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:57.465333 26299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:57.465543 26299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:57.465857 26299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:57.471077 26306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.471098 26308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:57.471163 26299 server_base.cc:1034] running on GCE node
W20260810 04:16:57.471133 26305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:57.471628 26299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:57.471975 26299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:57.473160 26299 hybrid_clock.cc:648] HybridClock initialized: now 1786335417473152 us; error 36 us; skew 500 ppm
I20260810 04:16:57.474678 26299 webserver.cc:460] Webserver started at http://127.15.216.66:35615/ using document root <none> and password file <none>
I20260810 04:16:57.474905 26299 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:57.474952 26299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:57.475055 26299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:57.476276 26299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "85f664095ea6499aa61eb4561a04ef16"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "4dcbda68bbf1f7ed665de7faa790be1a"
server_key_iv: "c1fd6e6d28f45f1d1127ed4b1b9bed46"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.476625 26299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "85f664095ea6499aa61eb4561a04ef16"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "4dcbda68bbf1f7ed665de7faa790be1a"
server_key_iv: "c1fd6e6d28f45f1d1127ed4b1b9bed46"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.478092 26299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:16:57.479055 26315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.479300 26299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:57.479421 26299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "85f664095ea6499aa61eb4561a04ef16"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "4dcbda68bbf1f7ed665de7faa790be1a"
server_key_iv: "c1fd6e6d28f45f1d1127ed4b1b9bed46"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.479516 26299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:57.494783 26299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:57.495147 26299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:57.495325 26299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:57.495608 26299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:57.495999 26299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:57.496061 26299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.496099 26299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:57.496131 26299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.502956 26299 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:38005
I20260810 04:16:57.503002 26432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:38005 every 8 connection(s)
I20260810 04:16:57.503386 26299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:57.508993 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.504228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:48849 (local address 127.15.216.126:46589)
0810 04:16:57.504299 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:57.504301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:57.504620 (+ 319us) server_negotiation.cc:407] Connection header received
0810 04:16:57.505130 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:57.505132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:57.505146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:57.505272 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:57.506019 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.506526 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.507573 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.507767 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.507825 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:57.508682 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:57.508683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:57.508683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:57.508688 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:57.508700 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:57.508703 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:57.508750 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:57.508913 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:16:57.508934 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:16:57.509239 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.504188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:46589 (local address 127.15.216.66:48849)
0810 04:16:57.504455 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:57.504501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:57.505021 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:57.505329 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:57.505332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:57.505341 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:57.505908 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:57.505913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.506618 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.506619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:57.507482 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:57.507484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.507852 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.507853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:57.507972 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:57.508521 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:57.508521 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:57.508562 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:57.508585 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:57.508818 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:57.508826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:57.508829 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:57.508902 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:16:57.508990 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260810 04:16:57.509931 26433 heartbeater.cc:347] Connected to a master server at 127.15.216.126:46589
I20260810 04:16:57.510040 26433 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:57.510224 26433 heartbeater.cc:511] Master 127.15.216.126:46589 requested a full tablet report, sending...
I20260810 04:16:57.510644 26097 ts_manager.cc:194] Registered new tserver with Master: 85f664095ea6499aa61eb4561a04ef16 (127.15.216.66:38005)
I20260810 04:16:57.510787 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26299
I20260810 04:16:57.510887 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:57.511114 16225 external_mini_cluster.cc:1428] Setting key 67e1f04291dbddc74c77cdd08dba9430
I20260810 04:16:57.511288 26097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.66:48849
I20260810 04:16:57.512142 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:46589
--builtin_ntp_servers=127.15.216.84:42229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 04:16:57.599335 26436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:57.599594 26436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:57.599630 26436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:57.600801 26436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:57.600872 26436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:57.600914 26436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:57.602314 26436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:42229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:57.602586 26436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:57.602890 26436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:57.608247 26441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.608301 26444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.608302 26442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:57.608347 26436 server_base.cc:1034] running on GCE node
I20260810 04:16:57.608721 26436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:57.608961 26436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:57.610132 26436 hybrid_clock.cc:648] HybridClock initialized: now 1786335417610104 us; error 40 us; skew 500 ppm
I20260810 04:16:57.611505 26436 webserver.cc:460] Webserver started at http://127.15.216.67:41261/ using document root <none> and password file <none>
I20260810 04:16:57.611790 26436 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:57.611846 26436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:57.612059 26436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:57.613266 26436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "cdfcfefc50bf48a9b62d751e28331ef3"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "95a207929a95a4c4572ae7524d4bbb53"
server_key_iv: "d7a8cfcddef0317431c03a85ce6d0323"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.613763 26436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "cdfcfefc50bf48a9b62d751e28331ef3"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "95a207929a95a4c4572ae7524d4bbb53"
server_key_iv: "d7a8cfcddef0317431c03a85ce6d0323"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.615337 26436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:57.616259 26450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.616516 26436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:57.616669 26436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "cdfcfefc50bf48a9b62d751e28331ef3"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "95a207929a95a4c4572ae7524d4bbb53"
server_key_iv: "d7a8cfcddef0317431c03a85ce6d0323"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.616766 26436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:57.629706 26436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:57.630046 26436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:57.630203 26436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:57.630462 26436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:57.630820 26436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:57.630880 26436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.630913 26436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:57.630959 26436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.636850 26436 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:45381
I20260810 04:16:57.636951 26569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:45381 every 8 connection(s)
I20260810 04:16:57.637279 26436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:57.638140 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26436
I20260810 04:16:57.638327 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:57.638586 16225 external_mini_cluster.cc:1428] Setting key bf882db8b0bf8eee7d00cd7867619179
I20260810 04:16:57.643002 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.638210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:58969 (local address 127.15.216.126:46589)
0810 04:16:57.638265 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:57.638267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:57.638609 (+ 342us) server_negotiation.cc:407] Connection header received
0810 04:16:57.639351 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:57.639353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:57.639367 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:57.639433 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:57.640197 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.640678 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.641678 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.641901 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.641974 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:57.642754 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:57.642755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:57.642756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:57.642760 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:57.642774 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:57.642778 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:57.642830 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:57.642916 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:16:57.642940 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:16:57.643208 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.638188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:46589 (local address 127.15.216.67:58969)
0810 04:16:57.638468 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:57.638514 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:57.639194 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:57.639502 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:57.639506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:57.639516 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:57.640093 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:57.640099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.640773 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.640773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:57.641550 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:57.641552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:57.641990 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:57.641990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:57.642099 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:57.642617 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:57.642617 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:57.642643 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:57.642665 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:57.642840 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:57.642844 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:57.642847 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:57.642900 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:16:57.642977 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
I20260810 04:16:57.643756 26570 heartbeater.cc:347] Connected to a master server at 127.15.216.126:46589
I20260810 04:16:57.643865 26570 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:57.644029 26570 heartbeater.cc:511] Master 127.15.216.126:46589 requested a full tablet report, sending...
I20260810 04:16:57.644515 26097 ts_manager.cc:194] Registered new tserver with Master: cdfcfefc50bf48a9b62d751e28331ef3 (127.15.216.67:45381)
I20260810 04:16:57.644979 26097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.67:58969
I20260810 04:16:57.650518 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:16:57.652572 26139 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:57.652021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.15.216.126:46589)
0810 04:16:57.652086 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:57.652088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:57.652293 (+ 205us) server_negotiation.cc:407] Connection header received
0810 04:16:57.652319 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:57.652320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:57.652331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:57.652378 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:57.652502 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49450: BlockingRecv error: recv got EOF from 127.0.0.1:49450 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:16:57.652652 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26160
I20260810 04:16:57.663257 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26299
I20260810 04:16:57.672477 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26436
I20260810 04:16:57.678177 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26064
2026-08-10T04:16:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (635 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T04:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:57Z Disabled control of system clock
I20260810 04:16:57.709231 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39561
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:39561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:16:57.800479 26581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:57.800720 26581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:57.800745 26581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:57.801949 26581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:57.801990 26581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:57.802003 26581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:57.802016 26581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:57.802029 26581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:57.803462 26581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:39561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:57.803680 26581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:57.804024 26581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:57.809422 26587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.809428 26586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.809430 26589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:57.809641 26581 server_base.cc:1034] running on GCE node
I20260810 04:16:57.810014 26581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:57.810329 26581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:57.811493 26581 hybrid_clock.cc:648] HybridClock initialized: now 1786335417811475 us; error 37 us; skew 500 ppm
I20260810 04:16:57.812852 26581 webserver.cc:460] Webserver started at http://127.15.216.126:40505/ using document root <none> and password file <none>
I20260810 04:16:57.813107 26581 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:57.813155 26581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:57.813316 26581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:57.814553 26581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "1177a2e8c655410fb09be0d6f646e99a"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "fcef4063004e5573ed1e095a0ab4d00b"
server_key_iv: "4fe0085c34626cc8719d4ca28bc53de0"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.814924 26581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "1177a2e8c655410fb09be0d6f646e99a"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "fcef4063004e5573ed1e095a0ab4d00b"
server_key_iv: "4fe0085c34626cc8719d4ca28bc53de0"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.816372 26581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:16:57.817307 26595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.817584 26581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:57.817766 26581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "1177a2e8c655410fb09be0d6f646e99a"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "fcef4063004e5573ed1e095a0ab4d00b"
server_key_iv: "4fe0085c34626cc8719d4ca28bc53de0"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.817852 26581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:57.822252 26581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:57.822561 26581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:57.822710 26581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:57.827198 26581 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:39561
I20260810 04:16:57.827234 26650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:39561 every 8 connection(s)
I20260810 04:16:57.827611 26581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:57.828302 26651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:57.831372 26651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a: Bootstrap starting.
I20260810 04:16:57.832029 26651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:57.832355 26651 log.cc:824] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:57.833094 26651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a: No bootstrap required, opened a new log
I20260810 04:16:57.834407 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26581
I20260810 04:16:57.834493 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:57.834697 16225 external_mini_cluster.cc:1428] Setting key d6c56a492a647f59c7342370209efa21
I20260810 04:16:57.835081 26651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1177a2e8c655410fb09be0d6f646e99a" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39561 } }
I20260810 04:16:57.835278 26651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:57.835336 26651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1177a2e8c655410fb09be0d6f646e99a, State: Initialized, Role: FOLLOWER
I20260810 04:16:57.835472 26651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1177a2e8c655410fb09be0d6f646e99a" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39561 } }
I20260810 04:16:57.835580 26651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:57.835615 26651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:57.835659 26651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:57.836476 26651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1177a2e8c655410fb09be0d6f646e99a" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39561 } }
I20260810 04:16:57.836593 26651 leader_election.cc:302] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1177a2e8c655410fb09be0d6f646e99a; no voters:
I20260810 04:16:57.836764 26651 leader_election.cc:288] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:57.836823 26656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:57.836982 26656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [term 1 LEADER]: Becoming Leader. State: Replica: 1177a2e8c655410fb09be0d6f646e99a, State: Running, Role: LEADER
I20260810 04:16:57.837105 26651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:57.837093 26656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1177a2e8c655410fb09be0d6f646e99a" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39561 } }
I20260810 04:16:57.837534 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.835369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.15.216.126:39561)
0810 04:16:57.835609 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:57.835619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:57.835653 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:16:57.835796 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:57.835804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:57.835860 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:57.836028 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:57.836807 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:57.836811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:57.836811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:57.836816 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:57.836865 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:57.836881 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:57.836959 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:57.837215 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 04:16:57.837286 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260810 04:16:57.837715 26657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1177a2e8c655410fb09be0d6f646e99a. Latest consensus state: current_term: 1 leader_uuid: "1177a2e8c655410fb09be0d6f646e99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1177a2e8c655410fb09be0d6f646e99a" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39561 } } }
I20260810 04:16:57.837809 26657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:57.838222 26662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:57.838409 26658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1177a2e8c655410fb09be0d6f646e99a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1177a2e8c655410fb09be0d6f646e99a" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39561 } } }
I20260810 04:16:57.838567 26658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:57.838804 26662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:57.840502 26662 catalog_manager.cc:1347] Generated new cluster ID: 8e94024f4a8b4baca4b423d7ad1a2c47
I20260810 04:16:57.840559 26662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:16:57.854202 26662 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:57.854398 26662 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:57.857003 26662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1177a2e8c655410fb09be0d6f646e99a: Generated new TSK 0
I20260810 04:16:57.857213 26662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:16:57.862910 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39561
--builtin_ntp_servers=127.15.216.84:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:16:57.962949 26675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:57.963186 26675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:57.963225 26675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:57.964419 26675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:57.964483 26675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:57.964529 26675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:57.966089 26675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:57.966358 26675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:57.966665 26675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:57.972162 26683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.972220 26680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:57.972124 26681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:57.972537 26675 server_base.cc:1034] running on GCE node
I20260810 04:16:57.972765 26675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:57.973050 26675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:57.974254 26675 hybrid_clock.cc:648] HybridClock initialized: now 1786335417974196 us; error 45 us; skew 500 ppm
I20260810 04:16:57.975857 26675 webserver.cc:460] Webserver started at http://127.15.216.65:40745/ using document root <none> and password file <none>
I20260810 04:16:57.976120 26675 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:57.976192 26675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:57.976372 26675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:57.977499 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "42a824a331444918ac165a6b85f62013"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "dc8d5757a12e2c770f57492f201ad126"
server_key_iv: "014ed38551b9f70cd631723e74a753c8"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.977962 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "42a824a331444918ac165a6b85f62013"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "dc8d5757a12e2c770f57492f201ad126"
server_key_iv: "014ed38551b9f70cd631723e74a753c8"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.979362 26675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:57.980250 26689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.980404 26675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:57.980499 26675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "42a824a331444918ac165a6b85f62013"
format_stamp: "Formatted at 2026-08-10 04:16:57 on dist-test-slave-g350"
server_key: "dc8d5757a12e2c770f57492f201ad126"
server_key_iv: "014ed38551b9f70cd631723e74a753c8"
server_key_version: "encryptionkey@0"
I20260810 04:16:57.980571 26675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:57.984159 26675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:57.984462 26675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:57.984601 26675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:57.984845 26675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:57.985240 26675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:57.985276 26675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.985296 26675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:57.985343 26675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:57.992776 26675 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:36789
I20260810 04:16:57.992871 26813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:36789 every 8 connection(s)
I20260810 04:16:57.993185 26675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:16:57.996388 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.994106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:50415 (local address 127.15.216.126:39561)
0810 04:16:57.994174 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:57.994176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:57.994431 (+ 255us) server_negotiation.cc:407] Connection header received
0810 04:16:57.994894 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:57.994895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:57.994903 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:57.994987 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:57.996168 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:57.996169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:57.996169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:57.996174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:57.996190 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:57.996194 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:57.996258 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:57.996319 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 04:16:57.996333 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:16:57.996655 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:57.994073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39561 (local address 127.15.216.65:50415)
0810 04:16:57.994255 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:57.994296 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:57.994782 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:57.995069 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:57.995073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:57.995081 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:57.995998 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:57.995998 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:57.996026 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:57.996052 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:57.996258 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:57.996260 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:57.996262 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:57.996365 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 04:16:57.996414 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260810 04:16:57.997275 26814 heartbeater.cc:347] Connected to a master server at 127.15.216.126:39561
I20260810 04:16:57.997414 26814 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:57.997579 26814 heartbeater.cc:511] Master 127.15.216.126:39561 requested a full tablet report, sending...
I20260810 04:16:57.998224 26613 ts_manager.cc:194] Registered new tserver with Master: 42a824a331444918ac165a6b85f62013 (127.15.216.65:36789)
I20260810 04:16:58.003182 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26675
I20260810 04:16:58.003262 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:58.003499 16225 external_mini_cluster.cc:1428] Setting key f6a77d7d8b04065d257d63050a30fb0c
I20260810 04:16:58.004572 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39561
--builtin_ntp_servers=127.15.216.84:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 04:16:58.090912 26817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:58.091128 26817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:58.091153 26817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:58.092335 26817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:58.092381 26817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:58.092407 26817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:16:58.093896 26817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:58.094164 26817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:58.094480 26817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:58.100015 26826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:58.100024 26823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:58.100059 26822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:58.100126 26817 server_base.cc:1034] running on GCE node
I20260810 04:16:58.100461 26817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:58.100790 26817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:58.101974 26817 hybrid_clock.cc:648] HybridClock initialized: now 1786335418101956 us; error 36 us; skew 500 ppm
I20260810 04:16:58.103262 26817 webserver.cc:460] Webserver started at http://127.15.216.66:44741/ using document root <none> and password file <none>
I20260810 04:16:58.103526 26817 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:58.103574 26817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:58.103667 26817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:58.104719 26817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "981d178098304112a6ee953b72bdf4aa"
format_stamp: "Formatted at 2026-08-10 04:16:58 on dist-test-slave-g350"
server_key: "7f5eb9b3cddfd3ca0707109bbcd3aad9"
server_key_iv: "ed9994a0f5768e7c89383e856c6b4ecf"
server_key_version: "encryptionkey@0"
I20260810 04:16:58.105016 26817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "981d178098304112a6ee953b72bdf4aa"
format_stamp: "Formatted at 2026-08-10 04:16:58 on dist-test-slave-g350"
server_key: "7f5eb9b3cddfd3ca0707109bbcd3aad9"
server_key_iv: "ed9994a0f5768e7c89383e856c6b4ecf"
server_key_version: "encryptionkey@0"
I20260810 04:16:58.106596 26817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:58.107450 26832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:58.107765 26817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:58.107858 26817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "981d178098304112a6ee953b72bdf4aa"
format_stamp: "Formatted at 2026-08-10 04:16:58 on dist-test-slave-g350"
server_key: "7f5eb9b3cddfd3ca0707109bbcd3aad9"
server_key_iv: "ed9994a0f5768e7c89383e856c6b4ecf"
server_key_version: "encryptionkey@0"
I20260810 04:16:58.107975 26817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:58.111670 26817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:58.111967 26817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:58.112102 26817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:58.112329 26817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:58.112646 26817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:58.112700 26817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:58.112741 26817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:58.112767 26817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:58.119171 26817 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:45681
I20260810 04:16:58.119320 26952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:45681 every 8 connection(s)
I20260810 04:16:58.119626 26817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:16:58.120584 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26817
I20260810 04:16:58.120692 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:16:58.120893 16225 external_mini_cluster.cc:1428] Setting key 55749399e7f5f9e02d2d3ab196f980f3
I20260810 04:16:58.121928 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39561
--builtin_ntp_servers=127.15.216.84:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 04:16:58.123183 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:58.120478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:45777 (local address 127.15.216.126:39561)
0810 04:16:58.120525 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:58.120527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:58.120800 (+ 273us) server_negotiation.cc:407] Connection header received
0810 04:16:58.121394 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:58.121395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:58.121401 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:58.121451 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:58.122944 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:58.122946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:58.122947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:58.122952 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:58.122969 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:58.122973 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:58.123022 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:58.123097 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 04:16:58.123112 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:16:58.123431 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:58.120436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39561 (local address 127.15.216.66:45777)
0810 04:16:58.120639 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:58.120674 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:58.121189 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:58.121521 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:58.121529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:58.121537 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:58.122774 (+ 1237us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:58.122775 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:58.122814 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:58.122852 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:58.123029 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:58.123035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:58.123038 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:16:58.123109 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:16:58.123172 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260810 04:16:58.123953 26953 heartbeater.cc:347] Connected to a master server at 127.15.216.126:39561
I20260810 04:16:58.124071 26953 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:58.124228 26953 heartbeater.cc:511] Master 127.15.216.126:39561 requested a full tablet report, sending...
I20260810 04:16:58.124672 26613 ts_manager.cc:194] Registered new tserver with Master: 981d178098304112a6ee953b72bdf4aa (127.15.216.66:45681)
W20260810 04:16:58.209713 26956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:58.209933 26956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:58.209959 26956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:58.211266 26956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:58.211319 26956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:58.211350 26956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:16:58.212805 26956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:39561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:58.213016 26956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:58.213286 26956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:58.218750 26961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:58.218850 26962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:58.218788 26964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:58.218992 26956 server_base.cc:1034] running on GCE node
I20260810 04:16:58.219240 26956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:58.219513 26956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:58.220682 26956 hybrid_clock.cc:648] HybridClock initialized: now 1786335418220651 us; error 38 us; skew 500 ppm
I20260810 04:16:58.222294 26956 webserver.cc:460] Webserver started at http://127.15.216.67:43279/ using document root <none> and password file <none>
I20260810 04:16:58.222581 26956 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:58.222653 26956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:58.222826 26956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:58.224006 26956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "893663b386b74555b2910800b70536ed"
format_stamp: "Formatted at 2026-08-10 04:16:58 on dist-test-slave-g350"
server_key: "c322d725346a3bffe541bc6845e9a0b4"
server_key_iv: "43ab154a1c11bfd2afedde2e0f9d57a6"
server_key_version: "encryptionkey@0"
I20260810 04:16:58.224407 26956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "893663b386b74555b2910800b70536ed"
format_stamp: "Formatted at 2026-08-10 04:16:58 on dist-test-slave-g350"
server_key: "c322d725346a3bffe541bc6845e9a0b4"
server_key_iv: "43ab154a1c11bfd2afedde2e0f9d57a6"
server_key_version: "encryptionkey@0"
I20260810 04:16:58.225971 26956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 04:16:58.226946 26972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:58.227190 26956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:58.227299 26956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "893663b386b74555b2910800b70536ed"
format_stamp: "Formatted at 2026-08-10 04:16:58 on dist-test-slave-g350"
server_key: "c322d725346a3bffe541bc6845e9a0b4"
server_key_iv: "43ab154a1c11bfd2afedde2e0f9d57a6"
server_key_version: "encryptionkey@0"
I20260810 04:16:58.227391 26956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:58.230973 26956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:58.231256 26956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:58.231401 26956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:58.231616 26956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:58.231927 26956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:58.231961 26956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:58.232012 26956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:58.232038 26956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:58.238520 26956 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:39869
I20260810 04:16:58.238564 27089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:39869 every 8 connection(s)
I20260810 04:16:58.238912 26956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:16:58.241997 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:58.239727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:49075 (local address 127.15.216.126:39561)
0810 04:16:58.239799 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:58.239801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:58.240077 (+ 276us) server_negotiation.cc:407] Connection header received
0810 04:16:58.240540 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:58.240541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:58.240547 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:58.240599 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:58.241755 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:58.241757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:58.241757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:16:58.241761 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:58.241776 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:16:58.241780 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:58.241839 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:58.241927 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:16:58.241940 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:16:58.242172 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:58.239727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39561 (local address 127.15.216.67:49075)
0810 04:16:58.239952 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:58.239986 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:58.240451 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:58.240651 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:58.240654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:58.240662 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:58.241546 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:16:58.241546 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:58.241573 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:16:58.241651 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:58.241853 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:58.241858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:58.241860 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:16:58.241920 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:16:58.241970 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260810 04:16:58.242722 27090 heartbeater.cc:347] Connected to a master server at 127.15.216.126:39561
I20260810 04:16:58.242816 27090 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:58.242930 27090 heartbeater.cc:511] Master 127.15.216.126:39561 requested a full tablet report, sending...
I20260810 04:16:58.243243 26613 ts_manager.cc:194] Registered new tserver with Master: 893663b386b74555b2910800b70536ed (127.15.216.67:39869)
I20260810 04:16:58.248751 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 26956
I20260810 04:16:58.248845 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:16:58.249069 16225 external_mini_cluster.cc:1428] Setting key e908fd0f1e4011d5cf6b96426fc38a9e
I20260810 04:16:58.250242 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 04:16:58.252177 26655 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:16:58.251742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.15.216.126:39561)
0810 04:16:58.251808 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:58.251810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:58.251936 (+ 126us) server_negotiation.cc:407] Connection header received
0810 04:16:58.251960 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:58.251960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:58.251965 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:58.252024 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:58.252123 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38038: BlockingRecv error: recv got EOF from 127.0.0.1:38038 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:16:58.252238 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26675
I20260810 04:16:58.258673 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26817
I20260810 04:16:58.269286 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26956
I20260810 04:16:58.275000 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 26581
2026-08-10T04:16:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (598 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:16:58 dist-test-slave-g350 krb5kdc[27100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:16:58 dist-test-slave-g350 krb5kdc[27100](info): set up 2 sockets
Aug 10 04:16:58 dist-test-slave-g350 krb5kdc[27100](info): commencing operation
krb5kdc: starting...
W20260810 04:16:59.404377 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:16:59 dist-test-slave-g350 krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:16:59.545234 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:37821
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:59.634286 27115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:59.634553 27115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:59.634591 27115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:59.635764 27115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:16:59.635841 27115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:59.635869 27115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:59.635900 27115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:16:59.635950 27115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:16:59.637495 27115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:37821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:59.637820 27115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:59.638068 27115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:59.643590 27121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:59.643599 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:59.643635 27125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:59.643867 27115 server_base.cc:1034] running on GCE node
I20260810 04:16:59.644165 27115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:59.644497 27115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:59.645745 27115 hybrid_clock.cc:648] HybridClock initialized: now 1786335419645687 us; error 46 us; skew 500 ppm
Aug 10 04:16:59 dist-test-slave-g350 krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335419, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:59.648649 27115 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:16:59.649142 27115 webserver.cc:460] Webserver started at http://127.15.216.126:39379/ using document root <none> and password file <none>
I20260810 04:16:59.649402 27115 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:59.649451 27115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:59.649641 27115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:59.650771 27115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "487e0d12d747402b8d47f49199a6bf38"
format_stamp: "Formatted at 2026-08-10 04:16:59 on dist-test-slave-g350"
server_key: "4878d2319bdd0e8e3bc361744bace5dd"
server_key_iv: "7ef298c08d6703612f79de7150d97363"
server_key_version: "encryptionkey@0"
I20260810 04:16:59.651153 27115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "487e0d12d747402b8d47f49199a6bf38"
format_stamp: "Formatted at 2026-08-10 04:16:59 on dist-test-slave-g350"
server_key: "4878d2319bdd0e8e3bc361744bace5dd"
server_key_iv: "7ef298c08d6703612f79de7150d97363"
server_key_version: "encryptionkey@0"
I20260810 04:16:59.652733 27115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:16:59.653719 27132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:59.653941 27115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:16:59.654042 27115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "487e0d12d747402b8d47f49199a6bf38"
format_stamp: "Formatted at 2026-08-10 04:16:59 on dist-test-slave-g350"
server_key: "4878d2319bdd0e8e3bc361744bace5dd"
server_key_iv: "7ef298c08d6703612f79de7150d97363"
server_key_version: "encryptionkey@0"
I20260810 04:16:59.654143 27115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:59.689833 27115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:59.692121 27115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:59.692315 27115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:59.696810 27115 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:37821
I20260810 04:16:59.696908 27184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:37821 every 8 connection(s)
I20260810 04:16:59.697281 27115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:16:59.698277 27185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:16:59.700767 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38: Bootstrap starting.
I20260810 04:16:59.701412 27185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38: Neither blocks nor log segments found. Creating new log.
I20260810 04:16:59.701774 27185 log.cc:824] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38: Log is configured to *not* fsync() on all Append() calls
I20260810 04:16:59.702657 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 27115
I20260810 04:16:59.702701 27185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38: No bootstrap required, opened a new log
I20260810 04:16:59.702766 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:16:59.703016 16225 external_mini_cluster.cc:1428] Setting key 6252f81bb1f724a411e94b5e6186cff7
I20260810 04:16:59.705989 27185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487e0d12d747402b8d47f49199a6bf38" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37821 } }
I20260810 04:16:59.706187 27185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:16:59.706228 27185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487e0d12d747402b8d47f49199a6bf38, State: Initialized, Role: FOLLOWER
I20260810 04:16:59.706406 27185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487e0d12d747402b8d47f49199a6bf38" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37821 } }
I20260810 04:16:59.706491 27185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:16:59.706516 27185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:16:59.706593 27185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:16:59.707321 27185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487e0d12d747402b8d47f49199a6bf38" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37821 } }
I20260810 04:16:59.707439 27185 leader_election.cc:302] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 487e0d12d747402b8d47f49199a6bf38; no voters:
I20260810 04:16:59.707602 27185 leader_election.cc:288] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:16:59.707686 27190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:16:59.707876 27185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:16:59.707891 27190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [term 1 LEADER]: Becoming Leader. State: Replica: 487e0d12d747402b8d47f49199a6bf38, State: Running, Role: LEADER
I20260810 04:16:59.708020 27190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487e0d12d747402b8d47f49199a6bf38" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37821 } }
I20260810 04:16:59.709106 27192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487e0d12d747402b8d47f49199a6bf38. Latest consensus state: current_term: 1 leader_uuid: "487e0d12d747402b8d47f49199a6bf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487e0d12d747402b8d47f49199a6bf38" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37821 } } }
I20260810 04:16:59.709343 27192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:59.709781 27191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "487e0d12d747402b8d47f49199a6bf38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487e0d12d747402b8d47f49199a6bf38" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37821 } } }
I20260810 04:16:59.709879 27191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38 [sys.catalog]: This master's current role is: LEADER
I20260810 04:16:59.710014 27203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:16:59.710796 27203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:16:59.712558 27203 catalog_manager.cc:1347] Generated new cluster ID: ef5c95e9806c4066a88b9b83126832ef
I20260810 04:16:59.712612 27203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:16:59 dist-test-slave-g350 krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:59.717406 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:59.703574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.15.216.126:37821)
0810 04:16:59.703772 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:59.703782 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:59.703825 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:16:59.705944 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:59.705954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:59.706027 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:59.706192 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:59.707059 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:59.708517 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:59.709992 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:59.710563 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:59.710696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:59.713542 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:59.713546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:59.713547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:59.713559 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:59.715419 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:59.715832 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:59.715835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:59.715835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:59.715932 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:59.716195 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:59.716196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:59.716197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:59.716451 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:59.716651 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:59.717023 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 04:16:59.717179 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260810 04:16:59.720731 27203 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:16:59.721426 27203 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:16:59.728018 27203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38: Generated new TSK 0
I20260810 04:16:59.728363 27203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:16:59.769237 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37821
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:16:59.868292 27213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:16:59.868564 27213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:16:59.868606 27213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:16:59.869885 27213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:16:59.869951 27213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:16:59.869999 27213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:16:59.871543 27213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:16:59.871822 27213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:16:59.872066 27213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:16:59.878285 27222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:59.878253 27219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:16:59.878273 27220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:16:59.878546 27213 server_base.cc:1034] running on GCE node
I20260810 04:16:59.878836 27213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:16:59.879104 27213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:16:59.880273 27213 hybrid_clock.cc:648] HybridClock initialized: now 1786335419880216 us; error 70 us; skew 500 ppm
Aug 10 04:16:59 dist-test-slave-g350 krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335419, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:16:59.883481 27213 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:16:59.883935 27213 webserver.cc:460] Webserver started at http://127.15.216.65:34493/ using document root <none> and password file <none>
I20260810 04:16:59.884141 27213 fs_manager.cc:359] Metadata directory not provided
I20260810 04:16:59.884238 27213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:16:59.884408 27213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:16:59.885696 27213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "7fa7a655d8a84bac9cd0080a37fe1d55"
format_stamp: "Formatted at 2026-08-10 04:16:59 on dist-test-slave-g350"
server_key: "10c6b153f494c8a014270a0a950c6c9c"
server_key_iv: "d941831c78fb8e74d08f54d55c84b86e"
server_key_version: "encryptionkey@0"
I20260810 04:16:59.886145 27213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "7fa7a655d8a84bac9cd0080a37fe1d55"
format_stamp: "Formatted at 2026-08-10 04:16:59 on dist-test-slave-g350"
server_key: "10c6b153f494c8a014270a0a950c6c9c"
server_key_iv: "d941831c78fb8e74d08f54d55c84b86e"
server_key_version: "encryptionkey@0"
I20260810 04:16:59.887687 27213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:16:59.888815 27229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:59.889057 27213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:16:59.889176 27213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "7fa7a655d8a84bac9cd0080a37fe1d55"
format_stamp: "Formatted at 2026-08-10 04:16:59 on dist-test-slave-g350"
server_key: "10c6b153f494c8a014270a0a950c6c9c"
server_key_iv: "d941831c78fb8e74d08f54d55c84b86e"
server_key_version: "encryptionkey@0"
I20260810 04:16:59.889263 27213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:16:59.919375 27213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:16:59.922008 27213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:16:59.922183 27213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:16:59.922464 27213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:16:59.922808 27213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:16:59.922864 27213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:59.922900 27213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:16:59.922932 27213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:16:59.929332 27213 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:33515
I20260810 04:16:59.929378 27353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:33515 every 8 connection(s)
I20260810 04:16:59.929778 27213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:16:59 dist-test-slave-g350 krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335419, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:16:59.937042 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 27213
I20260810 04:16:59.937156 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:16:59.937463 16225 external_mini_cluster.cc:1428] Setting key 3aec9b79debee28a3e0d2020bf2646b6
I20260810 04:16:59.940444 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:59.930566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:33417 (local address 127.15.216.126:37821)
0810 04:16:59.930664 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 04:16:59.930667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:16:59.930836 (+ 169us) server_negotiation.cc:407] Connection header received
0810 04:16:59.931438 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:16:59.931439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:16:59.931452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:16:59.931507 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:16:59.932505 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:59.933037 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:59.934106 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:16:59.934321 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:16:59.934411 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:59.936741 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:16:59.936743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:16:59.936743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:16:59.936752 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:16:59.938551 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:59.938979 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:59.938980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:59.938980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:59.939048 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:16:59.939389 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:16:59.939390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:16:59.939391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:16:59.939576 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:16:59.939660 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 04:16:59.940088 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 04:16:59.940260 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 04:16:59.940541 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:16:59.930460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:37821 (local address 127.15.216.65:33417)
0810 04:16:59.930697 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 04:16:59.930743 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:16:59.931322 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:16:59.931580 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:16:59.931583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:16:59.932188 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:16:59.932395 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:59.932399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:59.933144 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:59.933145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:59.934007 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:16:59.934010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:16:59.934426 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:16:59.934427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:16:59.934555 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:16:59.935076 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:16:59.935085 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:16:59.936638 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:16:59.938638 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:59.938645 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:59.938647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:59.938896 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:59.939136 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:16:59.939137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:16:59.939137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:16:59.939288 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:16:59.939675 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:16:59.939682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:16:59.939909 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 04:16:59.940033 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 04:16:59.940230 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":115,"threads_started":1}
I20260810 04:16:59.941401 27354 heartbeater.cc:347] Connected to a master server at 127.15.216.126:37821
I20260810 04:16:59.941498 27354 heartbeater.cc:464] Registering TS with master...
I20260810 04:16:59.941716 27354 heartbeater.cc:511] Master 127.15.216.126:37821 requested a full tablet report, sending...
I20260810 04:16:59.942338 27149 ts_manager.cc:194] Registered new tserver with Master: 7fa7a655d8a84bac9cd0080a37fe1d55 (127.15.216.65:33515)
I20260810 04:16:59.943331 27149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:33417
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:16:59.973472 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37821
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:17:00.068169 27359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:17:00.068414 27359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:17:00.068442 27359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:17:00.069687 27359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:17:00.069743 27359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:17:00.069773 27359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:17:00.071301 27359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:17:00.071547 27359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:17:00.071818 27359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:17:00.077765 27364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:00.077759 27365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:00.077917 27359 server_base.cc:1034] running on GCE node
W20260810 04:17:00.077859 27367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:00.078326 27359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:17:00.078604 27359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:17:00.079836 27359 hybrid_clock.cc:648] HybridClock initialized: now 1786335420079763 us; error 89 us; skew 500 ppm
Aug 10 04:17:00 dist-test-slave-g350 krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335420, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:00.082856 27359 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:17:00.083398 27359 webserver.cc:460] Webserver started at http://127.15.216.66:45931/ using document root <none> and password file <none>
I20260810 04:17:00.083669 27359 fs_manager.cc:359] Metadata directory not provided
I20260810 04:17:00.083742 27359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:17:00.083909 27359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:17:00.085162 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "9059a916b7824dc98d80204759473fec"
format_stamp: "Formatted at 2026-08-10 04:17:00 on dist-test-slave-g350"
server_key: "1888928a5fd63e6e2d35b1aa04c43b36"
server_key_iv: "4935e8cb349b458f25151c995d7310d1"
server_key_version: "encryptionkey@0"
I20260810 04:17:00.085542 27359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "9059a916b7824dc98d80204759473fec"
format_stamp: "Formatted at 2026-08-10 04:17:00 on dist-test-slave-g350"
server_key: "1888928a5fd63e6e2d35b1aa04c43b36"
server_key_iv: "4935e8cb349b458f25151c995d7310d1"
server_key_version: "encryptionkey@0"
I20260810 04:17:00.087015 27359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:17:00.087895 27374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:00.088101 27359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:17:00.088224 27359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "9059a916b7824dc98d80204759473fec"
format_stamp: "Formatted at 2026-08-10 04:17:00 on dist-test-slave-g350"
server_key: "1888928a5fd63e6e2d35b1aa04c43b36"
server_key_iv: "4935e8cb349b458f25151c995d7310d1"
server_key_version: "encryptionkey@0"
I20260810 04:17:00.088341 27359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:17:00.098477 27359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:17:00.101341 27359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:17:00.101529 27359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:17:00.101872 27359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:17:00.102283 27359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:17:00.102342 27359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:00.102386 27359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:17:00.102414 27359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:00.109475 27359 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:38539
I20260810 04:17:00.109520 27495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:38539 every 8 connection(s)
I20260810 04:17:00.109941 27359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:17:00.110925 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 27359
I20260810 04:17:00.111033 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:17:00.111269 16225 external_mini_cluster.cc:1428] Setting key 32a2b8a075fc1444071f9b802eee111c
Aug 10 04:17:00 dist-test-slave-g350 krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335420, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
I20260810 04:17:00.122030 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.110717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:41233 (local address 127.15.216.126:37821)
0810 04:17:00.110794 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.110797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.111082 (+ 285us) server_negotiation.cc:407] Connection header received
0810 04:17:00.111774 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.111775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.111786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.111856 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:00.113268 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.114148 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.115743 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.115978 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.116049 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.118772 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:00.118774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:00.118774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:00.118783 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:00.120340 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:00.120808 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:00.120809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:00.120810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:00.120902 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:00.121194 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:00.121195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:00.121195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:00.121386 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:00.121469 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.121832 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.121931 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:17:00.122264 27498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.110714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:37821 (local address 127.15.216.66:41233)
0810 04:17:00.110906 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.110957 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:00.111668 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:00.111943 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:00.111948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:00.112860 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:17:00.113150 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.113156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.114279 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.114281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.115608 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.115612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.116080 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.116081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.116247 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.116826 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:17:00.116837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:17:00.118656 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:17:00.120464 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:00.120473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:00.120476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:00.120723 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:00.120986 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:00.120987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:00.120987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:00.121119 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:00.121471 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:17:00.121480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:17:00.121691 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 04:17:00.121812 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 04:17:00.121991 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260810 04:17:00.122843 27496 heartbeater.cc:347] Connected to a master server at 127.15.216.126:37821
I20260810 04:17:00.122974 27496 heartbeater.cc:464] Registering TS with master...
I20260810 04:17:00.123180 27496 heartbeater.cc:511] Master 127.15.216.126:37821 requested a full tablet report, sending...
I20260810 04:17:00.123728 27149 ts_manager.cc:194] Registered new tserver with Master: 9059a916b7824dc98d80204759473fec (127.15.216.66:38539)
I20260810 04:17:00.124094 27149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:41233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:17:00.147195 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37821
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:17:00.241677 27501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:17:00.241916 27501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:17:00.241945 27501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:17:00.243137 27501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:17:00.243175 27501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:17:00.243201 27501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:17:00.244676 27501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:17:00.244882 27501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:17:00.245069 27501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:17:00.251299 27506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:00.251362 27507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:00.251708 27501 server_base.cc:1034] running on GCE node
W20260810 04:17:00.251708 27509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:00.252050 27501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:17:00.252334 27501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:17:00.253898 27501 hybrid_clock.cc:648] HybridClock initialized: now 1786335420253720 us; error 139 us; skew 500 ppm
Aug 10 04:17:00 dist-test-slave-g350 krb5kdc[27100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335420, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:00.256971 27501 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:17:00.257655 27501 webserver.cc:460] Webserver started at http://127.15.216.67:42699/ using document root <none> and password file <none>
I20260810 04:17:00.257952 27501 fs_manager.cc:359] Metadata directory not provided
I20260810 04:17:00.258029 27501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:17:00.258183 27501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:17:00.259424 27501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "68893fae21944789a4a447beff8bdf48"
format_stamp: "Formatted at 2026-08-10 04:17:00 on dist-test-slave-g350"
server_key: "d5315e328ebae9ce2e4883f193be7db5"
server_key_iv: "921b04cfadbeb67ca26050fc132b18c1"
server_key_version: "encryptionkey@0"
I20260810 04:17:00.259855 27501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "68893fae21944789a4a447beff8bdf48"
format_stamp: "Formatted at 2026-08-10 04:17:00 on dist-test-slave-g350"
server_key: "d5315e328ebae9ce2e4883f193be7db5"
server_key_iv: "921b04cfadbeb67ca26050fc132b18c1"
server_key_version: "encryptionkey@0"
I20260810 04:17:00.261513 27501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260810 04:17:00.262605 27516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:00.262981 27501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:17:00.263111 27501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "68893fae21944789a4a447beff8bdf48"
format_stamp: "Formatted at 2026-08-10 04:17:00 on dist-test-slave-g350"
server_key: "d5315e328ebae9ce2e4883f193be7db5"
server_key_iv: "921b04cfadbeb67ca26050fc132b18c1"
server_key_version: "encryptionkey@0"
I20260810 04:17:00.263203 27501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:17:00.285400 27501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:17:00.287992 27501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:17:00.288177 27501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:17:00.288462 27501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:17:00.288807 27501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:17:00.288866 27501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:00.288908 27501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:17:00.288938 27501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:00.296010 27501 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:42301
I20260810 04:17:00.296063 27638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:42301 every 8 connection(s)
I20260810 04:17:00.296412 27501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:17:00 dist-test-slave-g350 krb5kdc[27100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335420, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:17:00.305166 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 27501
I20260810 04:17:00.305276 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:17:00.305552 16225 external_mini_cluster.cc:1428] Setting key ff1b7418a490c3e40462a9dbb994579f
I20260810 04:17:00.308476 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.297174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:51061 (local address 127.15.216.126:37821)
0810 04:17:00.297264 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.297266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.297573 (+ 307us) server_negotiation.cc:407] Connection header received
0810 04:17:00.298153 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.298154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.298165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.298225 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:00.299415 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.300016 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.301104 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.301313 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.301383 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.304472 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:00.304474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:00.304475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:00.304488 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:00.306655 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:00.307211 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:00.307213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:00.307213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:00.307322 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:00.307640 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:00.307641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:00.307641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:00.307912 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:00.307989 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.308311 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.308402 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:17:00.309082 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.297140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:37821 (local address 127.15.216.67:51061)
0810 04:17:00.297427 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.297467 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:00.298007 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:00.298277 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:00.298280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:00.298881 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:17:00.299091 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.299096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.300150 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.300151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.300997 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.301000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.301546 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.301548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.301920 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.302576 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:17:00.302586 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:17:00.304230 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:17:00.306820 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:00.306828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:00.306831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:00.307109 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:00.307414 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:00.307414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:00.307415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:00.307532 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:00.307988 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:17:00.307994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:17:00.308140 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 04:17:00.308236 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 04:17:00.308752 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":123,"threads_started":1}
I20260810 04:17:00.310086 27639 heartbeater.cc:347] Connected to a master server at 127.15.216.126:37821
I20260810 04:17:00.310269 27639 heartbeater.cc:464] Registering TS with master...
I20260810 04:17:00.310503 27639 heartbeater.cc:511] Master 127.15.216.126:37821 requested a full tablet report, sending...
I20260810 04:17:00.311100 27149 ts_manager.cc:194] Registered new tserver with Master: 68893fae21944789a4a447beff8bdf48 (127.15.216.67:42301)
I20260810 04:17:00.311509 27149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:51061
I20260810 04:17:00.318718 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:17:00.327924 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.320188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.15.216.126:37821)
0810 04:17:00.320274 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.320277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.320413 (+ 136us) server_negotiation.cc:407] Connection header received
0810 04:17:00.320499 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.320500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.320512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.320565 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:00.321165 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.321732 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.322464 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.322670 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.322745 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.324402 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:00.324405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:00.324405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:00.324417 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:00.326564 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:00.327001 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:00.327002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:00.327003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:00.327066 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:00.327294 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:00.327295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:00.327295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:00.327463 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:00.327580 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.327719 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.327808 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 04:17:00.330590 27149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:17:00.332363 27149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:17:00.340026 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.335547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.15.216.67:42301)
0810 04:17:00.336409 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.336414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.336425 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:17:00.336469 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.336472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.336539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.336667 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:00.336754 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.338031 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.339133 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.339669 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.339742 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.339796 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.339895 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.339937 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":126,"threads_started":1}
I20260810 04:17:00.340001 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.335619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38539 (local address 127.0.0.1:40694)
0810 04:17:00.336034 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.336045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:00.336109 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:00.336488 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:00.336489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:00.336491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:00.336536 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.336536 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.337468 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.337470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.338577 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.338580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.339655 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.339655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.339736 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.339752 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 04:17:00.339820 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:17:00.339886 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":92,"threads_started":1}
I20260810 04:17:00.340001 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.335369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:42301 (local address 127.0.0.1:33484)
0810 04:17:00.335577 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.335620 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:00.335695 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:00.336666 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:00.336667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:00.336668 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:00.336708 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.336709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.338143 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.338144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.339033 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.339035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.339752 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.339753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.339816 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.339820 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:17:00.339873 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:17:00.339905 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":30,"thread_start_us":85,"threads_started":1}
I20260810 04:17:00.340737 27654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.335406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33515 (local address 127.0.0.1:47612)
0810 04:17:00.335766 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.335777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:00.335820 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:00.336220 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:00.336224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:00.336226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:00.336306 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.336308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.338308 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.338309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.339502 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.339505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.340488 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.340489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.340577 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.340582 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:17:00.340634 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:17:00.340676 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":40,"thread_start_us":85,"threads_started":1}
I20260810 04:17:00.340998 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.335829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.15.216.65:33515)
0810 04:17:00.336035 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.336039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.336051 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:17:00.336099 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.336102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.336120 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.336208 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:00.336397 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.337777 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.339593 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.340331 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.340401 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.340454 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.340867 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.340921 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":109,"threads_started":1}
I20260810 04:17:00.341078 27568 tablet_service.cc:1467] Processing CreateTablet for tablet 72cb0094730043f3841ee28190d63f7c (DEFAULT_TABLE table=test-table [id=e24f2ef3c34b40f39c5de4f0102524c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:17:00.341529 27568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cb0094730043f3841ee28190d63f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:17:00.341739 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.335760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.15.216.66:38539)
0810 04:17:00.336018 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.336025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.336104 (+ 79us) server_negotiation.cc:407] Connection header received
0810 04:17:00.336267 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.336270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.336287 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.336401 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:00.336617 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.337360 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.338675 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.339467 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.339547 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.339614 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.340248 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.340313 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20260810 04:17:00.341713 27282 tablet_service.cc:1467] Processing CreateTablet for tablet 72cb0094730043f3841ee28190d63f7c (DEFAULT_TABLE table=test-table [id=e24f2ef3c34b40f39c5de4f0102524c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:17:00.342020 27282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cb0094730043f3841ee28190d63f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:17:00.342329 27427 tablet_service.cc:1467] Processing CreateTablet for tablet 72cb0094730043f3841ee28190d63f7c (DEFAULT_TABLE table=test-table [id=e24f2ef3c34b40f39c5de4f0102524c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:17:00.342691 27427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72cb0094730043f3841ee28190d63f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:17:00.345682 27658 tablet_bootstrap.cc:492] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Bootstrap starting.
I20260810 04:17:00.345747 27660 tablet_bootstrap.cc:492] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Bootstrap starting.
I20260810 04:17:00.345933 27659 tablet_bootstrap.cc:492] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: Bootstrap starting.
I20260810 04:17:00.346627 27658 tablet_bootstrap.cc:654] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Neither blocks nor log segments found. Creating new log.
I20260810 04:17:00.346793 27660 tablet_bootstrap.cc:654] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Neither blocks nor log segments found. Creating new log.
I20260810 04:17:00.346858 27659 tablet_bootstrap.cc:654] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: Neither blocks nor log segments found. Creating new log.
I20260810 04:17:00.347342 27658 log.cc:824] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Log is configured to *not* fsync() on all Append() calls
I20260810 04:17:00.347568 27660 log.cc:824] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Log is configured to *not* fsync() on all Append() calls
I20260810 04:17:00.347685 27659 log.cc:824] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: Log is configured to *not* fsync() on all Append() calls
I20260810 04:17:00.348272 27658 tablet_bootstrap.cc:492] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: No bootstrap required, opened a new log
I20260810 04:17:00.348356 27658 ts_tablet_manager.cc:1397] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:17:00.348420 27660 tablet_bootstrap.cc:492] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: No bootstrap required, opened a new log
I20260810 04:17:00.348477 27659 tablet_bootstrap.cc:492] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: No bootstrap required, opened a new log
I20260810 04:17:00.348495 27660 ts_tablet_manager.cc:1397] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:17:00.348536 27659 ts_tablet_manager.cc:1397] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:17:00.350255 27660 raft_consensus.cc:348] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.350436 27660 raft_consensus.cc:374] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:17:00.350484 27660 raft_consensus.cc:729] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68893fae21944789a4a447beff8bdf48, State: Initialized, Role: FOLLOWER
I20260810 04:17:00.350603 27660 consensus_queue.cc:260] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.350929 27660 ts_tablet_manager.cc:1428] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:17:00.350950 27639 heartbeater.cc:503] Master 127.15.216.126:37821 was elected leader, sending a full tablet report...
I20260810 04:17:00.351052 27658 raft_consensus.cc:348] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.351073 27659 raft_consensus.cc:348] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.351253 27659 raft_consensus.cc:374] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:17:00.351253 27658 raft_consensus.cc:374] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:17:00.351300 27659 raft_consensus.cc:729] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa7a655d8a84bac9cd0080a37fe1d55, State: Initialized, Role: FOLLOWER
I20260810 04:17:00.351300 27658 raft_consensus.cc:729] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9059a916b7824dc98d80204759473fec, State: Initialized, Role: FOLLOWER
I20260810 04:17:00.351446 27659 consensus_queue.cc:260] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.351508 27658 consensus_queue.cc:260] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.351811 27658 ts_tablet_manager.cc:1428] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:17:00.351810 27659 ts_tablet_manager.cc:1428] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:17:00.351823 27354 heartbeater.cc:503] Master 127.15.216.126:37821 was elected leader, sending a full tablet report...
I20260810 04:17:00.351948 27496 heartbeater.cc:503] Master 127.15.216.126:37821 was elected leader, sending a full tablet report...
W20260810 04:17:00.360608 27497 tablet.cc:2369] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:17:00.425998 27664 raft_consensus.cc:482] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:17:00.426151 27664 raft_consensus.cc:504] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.427114 27664 leader_election.cc:288] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fa7a655d8a84bac9cd0080a37fe1d55 (127.15.216.65:33515), 9059a916b7824dc98d80204759473fec (127.15.216.66:38539)
W20260810 04:17:00.430694 27355 tablet.cc:2369] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:17:00.430898 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.427608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38539 (local address 127.15.216.67:42581)
0810 04:17:00.427800 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.427811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:00.427867 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:00.428093 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:00.428096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:00.428098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:00.428213 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.428217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.428896 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.428897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.429932 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.429935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.430565 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.430566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.430683 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.430687 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:17:00.430751 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:17:00.430798 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":135,"threads_started":1}
I20260810 04:17:00.430984 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.427787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:42581 (local address 127.15.216.66:38539)
0810 04:17:00.427929 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.427932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.427943 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:17:00.427976 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.427977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.427993 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.428066 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:00.428301 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.428794 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.430065 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.430469 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.430555 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.430567 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.430852 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.430908 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 04:17:00.431145 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.427603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:41047 (local address 127.15.216.65:33515)
0810 04:17:00.427687 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.427691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.427708 (+ 17us) server_negotiation.cc:407] Connection header received
0810 04:17:00.427761 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.427762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.427776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.427896 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:00.428163 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.428757 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.430013 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.430565 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.430662 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.430676 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.430958 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.431067 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:17:00.431344 27642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.427428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33515 (local address 127.15.216.67:41047)
0810 04:17:00.427521 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.427536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:00.427592 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:00.427948 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:00.427950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:00.427953 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:00.428068 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.428071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.428866 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.428868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.429906 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:00.429910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.430674 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.430675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:00.430774 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.430778 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:17:00.430833 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 04:17:00.430880 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":1}
I20260810 04:17:00.431457 27448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cb0094730043f3841ee28190d63f7c" candidate_uuid: "68893fae21944789a4a447beff8bdf48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9059a916b7824dc98d80204759473fec" is_pre_election: true
I20260810 04:17:00.431658 27448 raft_consensus.cc:2454] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68893fae21944789a4a447beff8bdf48 in term 0.
I20260810 04:17:00.431674 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cb0094730043f3841ee28190d63f7c" candidate_uuid: "68893fae21944789a4a447beff8bdf48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" is_pre_election: true
I20260810 04:17:00.431876 27303 raft_consensus.cc:2454] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68893fae21944789a4a447beff8bdf48 in term 0.
I20260810 04:17:00.431937 27519 leader_election.cc:302] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68893fae21944789a4a447beff8bdf48, 9059a916b7824dc98d80204759473fec; no voters:
I20260810 04:17:00.432102 27664 raft_consensus.cc:2781] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:17:00.432165 27664 raft_consensus.cc:482] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:17:00.432195 27664 raft_consensus.cc:3037] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:17:00.432976 27664 raft_consensus.cc:504] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.433136 27664 leader_election.cc:288] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [CANDIDATE]: Term 1 election: Requested vote from peers 7fa7a655d8a84bac9cd0080a37fe1d55 (127.15.216.65:33515), 9059a916b7824dc98d80204759473fec (127.15.216.66:38539)
I20260810 04:17:00.433338 27303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cb0094730043f3841ee28190d63f7c" candidate_uuid: "68893fae21944789a4a447beff8bdf48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55"
I20260810 04:17:00.433358 27448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72cb0094730043f3841ee28190d63f7c" candidate_uuid: "68893fae21944789a4a447beff8bdf48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9059a916b7824dc98d80204759473fec"
I20260810 04:17:00.433454 27303 raft_consensus.cc:3037] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:17:00.433449 27448 raft_consensus.cc:3037] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:17:00.434274 27448 raft_consensus.cc:2454] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68893fae21944789a4a447beff8bdf48 in term 1.
I20260810 04:17:00.434345 27303 raft_consensus.cc:2454] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68893fae21944789a4a447beff8bdf48 in term 1.
I20260810 04:17:00.434468 27519 leader_election.cc:302] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68893fae21944789a4a447beff8bdf48, 9059a916b7824dc98d80204759473fec; no voters:
I20260810 04:17:00.434615 27664 raft_consensus.cc:2781] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:17:00.434861 27664 raft_consensus.cc:686] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 1 LEADER]: Becoming Leader. State: Replica: 68893fae21944789a4a447beff8bdf48, State: Running, Role: LEADER
I20260810 04:17:00.434983 27664 consensus_queue.cc:237] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } }
I20260810 04:17:00.435946 27148 catalog_manager.cc:5495] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68893fae21944789a4a447beff8bdf48 (127.15.216.67). New cstate: current_term: 1 leader_uuid: "68893fae21944789a4a447beff8bdf48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68893fae21944789a4a447beff8bdf48" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 } health_report { overall_health: UNKNOWN } } }
I20260810 04:17:00.461201 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.458056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.15.216.67:42301)
0810 04:17:00.458253 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:00.458255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:00.458265 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:17:00.458386 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:00.458387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:00.458407 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:00.458487 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:17:00.458727 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.459224 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.460137 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:00.460423 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:00.460495 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:00.460769 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:17:00.460907 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:17:00.460978 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:00.461045 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:17:00.461066 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260810 04:17:00.464133 27303 raft_consensus.cc:1261] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 1 FOLLOWER]: Refusing update from remote peer 68893fae21944789a4a447beff8bdf48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:17:00.464188 27448 raft_consensus.cc:1261] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 1 FOLLOWER]: Refusing update from remote peer 68893fae21944789a4a447beff8bdf48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:17:00.464602 27668 consensus_queue.cc:1035] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9059a916b7824dc98d80204759473fec" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:17:00.464775 27664 consensus_queue.cc:1035] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa7a655d8a84bac9cd0080a37fe1d55" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:17:00.468137 27672 mvcc.cc:205] Tried to move back new op lower bound from 7316829882217926656 to 7316829882102632448. Current Snapshot: MvccSnapshot[applied={T|T < 7316829882217926656}]
I20260810 04:17:00.470568 27148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:17:00.470664 27148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:17:00.471642 27148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 487e0d12d747402b8d47f49199a6bf38: Sending DeleteTablet for 3 replicas of tablet 72cb0094730043f3841ee28190d63f7c
I20260810 04:17:00.472064 27282 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cb0094730043f3841ee28190d63f7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:17:00 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:47612
I20260810 04:17:00.472069 27568 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cb0094730043f3841ee28190d63f7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:17:00 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:33484
I20260810 04:17:00.472119 27427 tablet_service.cc:1514] Processing DeleteTablet for tablet 72cb0094730043f3841ee28190d63f7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:17:00 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:40694
I20260810 04:17:00.472374 27680 tablet_replica.cc:330] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55: stopping tablet replica
I20260810 04:17:00.472432 27682 tablet_replica.cc:330] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: stopping tablet replica
I20260810 04:17:00.472553 27682 raft_consensus.cc:2229] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:17:00.472579 27680 raft_consensus.cc:2229] T 72cb0094730043f3841ee28190d63f7c P 7fa7a655d8a84bac9cd0080a37fe1d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:17:00.472671 27682 raft_consensus.cc:2258] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:17:00.472713 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 27213
I20260810 04:17:00.475535 27681 tablet_replica.cc:330] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: stopping tablet replica
I20260810 04:17:00.475692 27681 raft_consensus.cc:2229] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:17:00.475854 27681 raft_consensus.cc:2258] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:17:00.476447 27681 ts_tablet_manager.cc:1905] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:17:00.478650 27681 ts_tablet_manager.cc:1918] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:17:00.478756 27681 log.cc:1196] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/72cb0094730043f3841ee28190d63f7c
I20260810 04:17:00.479063 27681 ts_tablet_manager.cc:1939] T 72cb0094730043f3841ee28190d63f7c P 9059a916b7824dc98d80204759473fec: Deleting consensus metadata
I20260810 04:17:00.479569 27135 catalog_manager.cc:4841] TS 9059a916b7824dc98d80204759473fec (127.15.216.66:38539): tablet 72cb0094730043f3841ee28190d63f7c (table test-table [id=e24f2ef3c34b40f39c5de4f0102524c2]) successfully deleted
I20260810 04:17:00.481957 27682 ts_tablet_manager.cc:1905] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:17:00.484256 27682 ts_tablet_manager.cc:1918] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:17:00.484371 27682 log.cc:1196] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/72cb0094730043f3841ee28190d63f7c
W20260810 04:17:00.484623 27135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:33515 (error 108)
I20260810 04:17:00.484709 27682 ts_tablet_manager.cc:1939] T 72cb0094730043f3841ee28190d63f7c P 68893fae21944789a4a447beff8bdf48: Deleting consensus metadata
I20260810 04:17:00.485306 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:00.485144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33515 (local address 127.0.0.1:47616)
0810 04:17:00.485216 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:00.485268 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 04:17:00.485432 27135 catalog_manager.cc:4584] TS 7fa7a655d8a84bac9cd0080a37fe1d55 (127.15.216.65:33515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72cb0094730043f3841ee28190d63f7c: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33515: connect: Connection refused (error 111)
I20260810 04:17:00.485443 27133 catalog_manager.cc:4841] TS 68893fae21944789a4a447beff8bdf48 (127.15.216.67:42301): tablet 72cb0094730043f3841ee28190d63f7c (table test-table [id=e24f2ef3c34b40f39c5de4f0102524c2]) successfully deleted
I20260810 04:17:00.485860 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 27359
I20260810 04:17:00.494555 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 27501
I20260810 04:17:00.501384 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 27115
2026-08-10T04:17:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2224 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:17:00 dist-test-slave-g350 krb5kdc[27686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:17:00 dist-test-slave-g350 krb5kdc[27686](info): set up 2 sockets
Aug 10 04:17:00 dist-test-slave-g350 krb5kdc[27686](info): commencing operation
krb5kdc: starting...
W20260810 04:17:01.648562 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:17:01 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:17:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/rangeradmin_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/rangeradmin_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/rangerlookup_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/rangerlookup_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
I20260810 04:17:01.857843 16225 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrlBm0x/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/postgres -l logfile start
2026-08-10 04:17:04.969 UTC [27721] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 04:17:04.969 UTC [27721] LOG: listening on IPv4 address "127.15.216.84", port 45141
2026-08-10 04:17:04.974 UTC [27721] LOG: listening on Unix socket "/tmp/.s.PGSQL.45141"
2026-08-10 04:17:05.011 UTC [27724] LOG: database system was shut down at 2026-08-10 04:17:03 UTC
2026-08-10 04:17:05.016 UTC [27721] LOG: database system is ready to accept connections
I20260810 04:17:06.028847 16225 mini_postgres.cc:96] Postgres bound to 45141
2026-08-10 04:17:06.037 UTC [27733] FATAL: database "slave" does not exist
127.15.216.84:45141 - accepting connections
I20260810 04:17:06.038733 16225 mini_ranger.cc:161] Starting Ranger...
I20260810 04:17:06.056155 16225 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 04:17:06.602624 16225 mini_ranger.cc:87] Created ranger Postgres database
I20260810 04:17:06.602721 16225 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-admin
2026-08-10 04:17:06,957 [I] DB FLAVOR :POSTGRES
2026-08-10 04:17:06,957 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:17:06,957 [I] Checking connection..
2026-08-10 04:17:06,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 04:17:07,237 [I] Checking connection passed.
2026-08-10 04:17:07,238 [I] --------- Verifying version history table ---------
2026-08-10 04:17:07,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:17:07,510 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:17:07,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:17:07,773 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:17:07,773 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 04:17:07,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 04:17:08.042 UTC [27800] WARNING: there is no transaction in progress
2026-08-10 04:17:08,050 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 04:17:08,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:17:08,304 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 04:17:08,304 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:17:08,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 04:17:08,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 04:17:08,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g350.c.gcp-upstream.internal','N') ;"
2026-08-10 04:17:09,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:17:09,370 [I] Table x_portal_user does not exist in database ranger
2026-08-10 04:17:09,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:17:09,624 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 04:17:09,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:17:09,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:17:10,179 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 04:17:10,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 04:17:11.289 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.308 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.326 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.343 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.361 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.380 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.444 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.452 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.467 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.477 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.488 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.503 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.513 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.520 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.527 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:11.535 UTC [27915] WARNING: there is no transaction in progress
2026-08-10 04:17:12,279 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 04:17:12,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 04:17:12,566 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 04:17:12,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:17:12,828 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 04:17:12,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:17:13,089 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 04:17:13,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:17:13,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:17:13,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g350.c.gcp-upstream.internal';"
2026-08-10 04:17:13,891 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 04:17:13,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 04:17:14,134 [I] DB_PATCHES have already been applied
I20260810 04:17:14.141121 16225 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrlBm0x/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 04:17:14.141191 16225 mini_ranger.cc:193] Using host: 127.15.216.84
I20260810 04:17:14.145062 16225 mini_ranger.cc:227] Ranger admin URL: http://127.15.216.84:54733
Aug 10, 2026 4:17:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:17:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:17:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 4:17:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 4:17:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 4:17:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 4:17:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 04:17:15.310606 16225 mini_ranger.cc:160] Time spent starting Ranger: real 9.272s user 0.000s sys 0.012s
Aug 10, 2026 4:17:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 4:17:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 4:17:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 4:17:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 4:17:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 04:17:38.351137 16225 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/rangerkms_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/rangerkms_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.216.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.216.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 04:17:38 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 04:17:38.446403 16225 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 04:17:38.457468 16225 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 04:17:39.532407 16225 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 04:17:39.532507 16225 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms
I20260810 04:17:39.532527 16225 mini_ranger_kms.cc:227] Using postgres at 127.15.216.84:45141
2026-08-10 04:17:40,128 [I] DB FLAVOR :POSTGRES
2026-08-10 04:17:40,128 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:17:40,128 [I] Checking connection
2026-08-10 04:17:40,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 04:17:40,375 [I] connection success
2026-08-10 04:17:40,376 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 04:17:40,376 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 04:17:40,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 04:17:40,629 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 04:17:40,629 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:17:40,629 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 04:17:40,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:45141/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 04:17:40,924 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 04:17:41.461239 16225 mini_ranger_kms.cc:310] Created kms service
I20260810 04:17:41.676944 16225 mini_ranger_kms.cc:326] Created kudu user
I20260810 04:17:41.722720 16225 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 04:17:41 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:17:42.142026 16225 mini_ranger_kms.cc:384] Added ranger policy
I20260810 04:17:42.142149 16225 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 04:17:42.142170 16225 mini_ranger_kms.cc:240] Using host: 127.15.216.84
I20260810 04:17:42.146348 16225 mini_ranger_kms.cc:276] Ranger KMS PID: 28104
I20260810 04:17:42.146492 16225 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.216.84:46701
Aug 10, 2026 4:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 4:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 4:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 4:17:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 04:17:44 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786335464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.216.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:44.511706 16225 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.065s user 0.021s sys 0.258s
I20260810 04:17:44.511826 16225 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 04:17:44.511916 16225 mini_ranger_kms.cc:401] 127.15.216.84:46701/kms/v1/keys
Aug 10 04:17:44 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.216.84@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
Aug 10 04:17:45 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.216.84@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
Aug 10 04:17:47 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:17:47.610528 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45125
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45125
--ranger_config_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:46701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:17:47.695675 28158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:17:47.695906 28158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:17:47.695963 28158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:17:47.697113 28158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:17:47.697180 28158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:17:47.697206 28158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:17:47.697235 28158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:17:47.697268 28158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:17:47.698992 28158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.216.84:46701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45125
--ranger_config_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:17:47.699276 28158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:17:47.699504 28158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:17:47.704952 28163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:47.705127 28166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:47.704960 28164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:47.705403 28158 server_base.cc:1034] running on GCE node
I20260810 04:17:47.705696 28158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:17:47.706025 28158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:17:47.707196 28158 hybrid_clock.cc:648] HybridClock initialized: now 1786335467707175 us; error 40 us; skew 500 ppm
Aug 10 04:17:47 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335467, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:47.714596 28158 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:17:47.715161 28158 webserver.cc:460] Webserver started at http://127.15.216.126:42259/ using document root <none> and password file <none>
I20260810 04:17:47.715431 28158 fs_manager.cc:359] Metadata directory not provided
I20260810 04:17:47.715497 28158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:17:47.715672 28158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:17:47 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335467, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:17:47.776152 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "2ec61d8ccb4144c784b852abc61aefc4"
format_stamp: "Formatted at 2026-08-10 04:17:47 on dist-test-slave-g350"
server_key: "87726d4d6654b27cfeba211f14a6bc50"
server_key_iv: "e4d0f11ec07ede90811ee0ab735b631f"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:47.776624 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "2ec61d8ccb4144c784b852abc61aefc4"
format_stamp: "Formatted at 2026-08-10 04:17:47 on dist-test-slave-g350"
server_key: "87726d4d6654b27cfeba211f14a6bc50"
server_key_iv: "e4d0f11ec07ede90811ee0ab735b631f"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:47.778671 28158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.002s
I20260810 04:17:47.794768 28173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:47.795071 28158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:17:47.795231 28158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "2ec61d8ccb4144c784b852abc61aefc4"
format_stamp: "Formatted at 2026-08-10 04:17:47 on dist-test-slave-g350"
server_key: "87726d4d6654b27cfeba211f14a6bc50"
server_key_iv: "e4d0f11ec07ede90811ee0ab735b631f"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:47.795392 28158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:17:47.808094 28158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:17:47.810477 28158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:17:47.810658 28158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:17:47.816011 28158 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:45125
I20260810 04:17:47.816042 28228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:45125 every 8 connection(s)
I20260810 04:17:47.816566 28158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:17:47.817333 28229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:17:47.817389 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 28158
I20260810 04:17:47.817499 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:17:47.817715 16225 external_mini_cluster.cc:1428] Setting key ad5847674c7e9856d4900b353e8c967a
I20260810 04:17:47.821359 28229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4: Bootstrap starting.
I20260810 04:17:47.822142 28229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4: Neither blocks nor log segments found. Creating new log.
I20260810 04:17:47.822487 28229 log.cc:824] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4: Log is configured to *not* fsync() on all Append() calls
I20260810 04:17:47.824260 28229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4: No bootstrap required, opened a new log
Aug 10 04:17:47 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:17:47.827446 28229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec61d8ccb4144c784b852abc61aefc4" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:17:47.827644 28229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:17:47.827683 28229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec61d8ccb4144c784b852abc61aefc4, State: Initialized, Role: FOLLOWER
I20260810 04:17:47.827898 28229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec61d8ccb4144c784b852abc61aefc4" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:17:47.828019 28229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:17:47.828083 28229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:17:47.828143 28229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:17:47.828948 28229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec61d8ccb4144c784b852abc61aefc4" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:17:47.829079 28229 leader_election.cc:302] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ec61d8ccb4144c784b852abc61aefc4; no voters:
I20260810 04:17:47.829308 28229 leader_election.cc:288] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:17:47.829341 28234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:17:47.829705 28229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:17:47.830276 28234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [term 1 LEADER]: Becoming Leader. State: Replica: 2ec61d8ccb4144c784b852abc61aefc4, State: Running, Role: LEADER
I20260810 04:17:47.830456 28234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec61d8ccb4144c784b852abc61aefc4" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:17:47.831182 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:47.818282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.15.216.126:45125)
0810 04:17:47.818460 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:47.818470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:47.818501 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:17:47.819182 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:47.819192 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:47.819259 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:47.819414 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:47.820386 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:47.821865 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:47.823009 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:47.823616 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:47.823732 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:47.827260 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:47.827266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:47.827267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:47.827282 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:47.829406 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:47.829861 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:47.829865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:47.829865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:47.829966 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:47.830181 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:47.830182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:47.830182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:47.830505 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:47.830634 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:47.830843 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 04:17:47.830970 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260810 04:17:47.831413 28234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec61d8ccb4144c784b852abc61aefc4. Latest consensus state: current_term: 1 leader_uuid: "2ec61d8ccb4144c784b852abc61aefc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec61d8ccb4144c784b852abc61aefc4" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } } }
I20260810 04:17:47.831499 28234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [sys.catalog]: This master's current role is: LEADER
I20260810 04:17:47.832636 28235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ec61d8ccb4144c784b852abc61aefc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec61d8ccb4144c784b852abc61aefc4" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } } }
I20260810 04:17:47.832715 28235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4 [sys.catalog]: This master's current role is: LEADER
I20260810 04:17:47.832960 28229 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:17:47.833616 28249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:17:47.836285 28249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:17:47.839267 28249 catalog_manager.cc:1347] Generated new cluster ID: da43c3a93be24172a10fe1fb1cee1824
I20260810 04:17:47.839321 28249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:17:47.852079 28249 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:17:47.852931 28249 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:17:47.859042 28249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4: Generated new TSK 0
I20260810 04:17:47.859342 28249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:17:48.653196 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:46701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45125
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:17:48.747538 28280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:17:48.747812 28280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:17:48.747859 28280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:17:48.749219 28280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:17:48.749300 28280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:17:48.749351 28280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:17:48.751029 28280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.216.84:46701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:17:48.751338 28280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:17:48.751597 28280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:17:48.758677 28288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:48.758729 28285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:48.758729 28286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:48.758951 28280 server_base.cc:1034] running on GCE node
I20260810 04:17:48.759220 28280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:17:48.759536 28280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:17:48.760735 28280 hybrid_clock.cc:648] HybridClock initialized: now 1786335468760704 us; error 45 us; skew 500 ppm
Aug 10 04:17:48 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335468, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:48.763691 28280 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:17:48.764277 28280 webserver.cc:460] Webserver started at http://127.15.216.65:42587/ using document root <none> and password file <none>
I20260810 04:17:48.764559 28280 fs_manager.cc:359] Metadata directory not provided
I20260810 04:17:48.764631 28280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:17:48.764788 28280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:17:48 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335468, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:17:48.778754 28280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "e2ac0ca21ca446f0bc858f981fb918d7"
format_stamp: "Formatted at 2026-08-10 04:17:48 on dist-test-slave-g350"
server_key: "7ba687d04fb0a647bc148d994abef9a3"
server_key_iv: "8c8660ef5b5f91d7b3a00bc4980a2bef"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:48.779260 28280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "e2ac0ca21ca446f0bc858f981fb918d7"
format_stamp: "Formatted at 2026-08-10 04:17:48 on dist-test-slave-g350"
server_key: "7ba687d04fb0a647bc148d994abef9a3"
server_key_iv: "8c8660ef5b5f91d7b3a00bc4980a2bef"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:48.782054 28280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 04:17:48.797858 28296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:48.798122 28280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:17:48.798244 28280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "e2ac0ca21ca446f0bc858f981fb918d7"
format_stamp: "Formatted at 2026-08-10 04:17:48 on dist-test-slave-g350"
server_key: "7ba687d04fb0a647bc148d994abef9a3"
server_key_iv: "8c8660ef5b5f91d7b3a00bc4980a2bef"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:48.798352 28280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:17:48.808684 28280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:17:48.811335 28280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:17:48.811523 28280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:17:48.811797 28280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:17:48.812250 28280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:17:48.812301 28280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:48.812335 28280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:17:48.812383 28280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:48.820020 28280 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:32867
I20260810 04:17:48.820832 28280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:17:48.821624 28415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:32867 every 8 connection(s)
I20260810 04:17:48.830098 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 28280
I20260810 04:17:48.830245 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:17:48.830494 16225 external_mini_cluster.cc:1428] Setting key 518cadfa659a8c6d963ea7b36094d389
Aug 10 04:17:48 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335468, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:17:48.836328 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:48.822556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:50475 (local address 127.15.216.126:45125)
0810 04:17:48.825835 (+ 3279us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:48.825838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:48.825848 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:17:48.825876 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:48.825877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:48.825891 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:48.826007 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:48.826946 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:48.827747 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:48.829413 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:48.829686 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:48.829775 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:48.832627 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:48.832629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:48.832629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:48.832639 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:48.834715 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:48.835184 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:48.835186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:48.835186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:48.835258 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:48.835594 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:48.835595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:48.835596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:48.835724 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:48.835804 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:48.836119 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 04:17:48.836212 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3238,"thread_start_us":88,"threads_started":1}
I20260810 04:17:48.836517 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:48.822226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45125 (local address 127.15.216.65:50475)
0810 04:17:48.824733 (+ 2507us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:48.824812 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:48.825611 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:48.826154 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:48.826159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:48.826474 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:17:48.826810 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:48.826818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:48.827866 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:48.827868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:48.829185 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:48.829189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:48.829783 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:48.829784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:48.829959 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:48.830670 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:17:48.830681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:17:48.832515 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:17:48.834827 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:48.834835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:48.834838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:48.835074 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:48.835390 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:48.835391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:48.835391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:48.835520 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:48.835795 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:17:48.835810 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:17:48.835970 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 04:17:48.836086 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 04:17:48.836249 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2433,"thread_start_us":96,"threads_started":1}
I20260810 04:17:48.837011 28416 heartbeater.cc:347] Connected to a master server at 127.15.216.126:45125
I20260810 04:17:48.837114 28416 heartbeater.cc:464] Registering TS with master...
I20260810 04:17:48.837430 28416 heartbeater.cc:511] Master 127.15.216.126:45125 requested a full tablet report, sending...
I20260810 04:17:48.838053 28190 ts_manager.cc:194] Registered new tserver with Master: e2ac0ca21ca446f0bc858f981fb918d7 (127.15.216.65:32867)
I20260810 04:17:48.838773 28190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:50475
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:17:48.868963 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:46701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45125
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:17:48.995069 28425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:17:48.995386 28425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:17:48.995425 28425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:17:48.997125 28425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:17:48.997188 28425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:17:48.997229 28425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:17:48.999436 28425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.216.84:46701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:17:48.999907 28425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:17:49.000195 28425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:17:49.006943 28435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:49.006951 28433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:49.007234 28432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:49.007881 28425 server_base.cc:1034] running on GCE node
I20260810 04:17:49.008129 28425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:17:49.008490 28425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:17:49.009686 28425 hybrid_clock.cc:648] HybridClock initialized: now 1786335469009669 us; error 31 us; skew 500 ppm
Aug 10 04:17:49 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335469, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:49.013424 28425 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:17:49.014022 28425 webserver.cc:460] Webserver started at http://127.15.216.66:45351/ using document root <none> and password file <none>
I20260810 04:17:49.014285 28425 fs_manager.cc:359] Metadata directory not provided
I20260810 04:17:49.014349 28425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:17:49.014535 28425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:17:49 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335469, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:17:49.031361 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "3c86646d4ebc4fda915098d4ceca3c25"
format_stamp: "Formatted at 2026-08-10 04:17:49 on dist-test-slave-g350"
server_key: "2aa744b59ca51fc325d70a980375ac29"
server_key_iv: "bb1613bf4d4414e3b430dae15f9edd2f"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:49.032038 28425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "3c86646d4ebc4fda915098d4ceca3c25"
format_stamp: "Formatted at 2026-08-10 04:17:49 on dist-test-slave-g350"
server_key: "2aa744b59ca51fc325d70a980375ac29"
server_key_iv: "bb1613bf4d4414e3b430dae15f9edd2f"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:49.038061 28425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
Aug 10 04:17:49 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786335469, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:49.065773 28442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:49.066020 28425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260810 04:17:49.066172 28425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "3c86646d4ebc4fda915098d4ceca3c25"
format_stamp: "Formatted at 2026-08-10 04:17:49 on dist-test-slave-g350"
server_key: "2aa744b59ca51fc325d70a980375ac29"
server_key_iv: "bb1613bf4d4414e3b430dae15f9edd2f"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:49.066346 28425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:17:49.081698 28425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:17:49.084878 28425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:17:49.085107 28425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:17:49.085420 28425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:17:49.085904 28425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:17:49.085999 28425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:49.086066 28425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:17:49.086118 28425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:49.093297 28425 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:39235
I20260810 04:17:49.093874 28425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:17:49.094610 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 28425
I20260810 04:17:49.094738 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:17:49.094951 16225 external_mini_cluster.cc:1428] Setting key 008d6e9fb68f35e90ffd20b2295f8603
I20260810 04:17:49.098691 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:39235 every 8 connection(s)
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Aug 10 04:17:49 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335469, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:17:49.123718 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.104974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:36615 (local address 127.15.216.126:45125)
0810 04:17:49.105042 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.105045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.108815 (+ 3770us) server_negotiation.cc:407] Connection header received
0810 04:17:49.109648 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.109650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.109670 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.109708 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:49.111112 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.111978 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.116654 (+ 4676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.117004 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.117064 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.120101 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:49.120102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:49.120103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:49.120112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:49.121922 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:49.122492 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:49.122493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:49.122494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:49.122558 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:49.122841 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:49.122842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:49.122843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:49.122999 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:49.123053 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.123493 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.123594 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 04:17:49.124595 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.104609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45125 (local address 127.15.216.66:36615)
0810 04:17:49.108628 (+ 4019us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.108692 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:49.109476 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:49.109930 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:49.109935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:49.110203 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:17:49.110539 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.110546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.112321 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.112322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.113878 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.113882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.117184 (+ 3302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.117185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.117359 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.118080 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:17:49.118093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:17:49.119972 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:17:49.122031 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:49.122042 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:49.122045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:49.122390 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:49.122624 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:49.122625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:49.122625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:49.122763 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:49.123123 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:17:49.123130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:17:49.123312 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 04:17:49.124098 (+ 786us) client_negotiation.cc:240] Negotiation successful
0810 04:17:49.124342 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3955,"thread_start_us":113,"threads_started":1}
I20260810 04:17:49.129875 28565 heartbeater.cc:347] Connected to a master server at 127.15.216.126:45125
I20260810 04:17:49.129978 28565 heartbeater.cc:464] Registering TS with master...
I20260810 04:17:49.130223 28565 heartbeater.cc:511] Master 127.15.216.126:45125 requested a full tablet report, sending...
I20260810 04:17:49.130798 28190 ts_manager.cc:194] Registered new tserver with Master: 3c86646d4ebc4fda915098d4ceca3c25 (127.15.216.66:39235)
I20260810 04:17:49.131301 28190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:36615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:17:49.141507 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:46701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45125
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:17:49.246570 28570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:17:49.246817 28570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:17:49.246857 28570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:17:49.248116 28570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:17:49.248190 28570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:17:49.248237 28570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:17:49.250002 28570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.216.84:46701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:45125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:17:49.250293 28570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:17:49.250615 28570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:17:49.256718 28577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:49.257331 28576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:17:49.258136 28579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:17:49.258960 28570 server_base.cc:1034] running on GCE node
I20260810 04:17:49.259209 28570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:17:49.259565 28570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:17:49.260793 28570 hybrid_clock.cc:648] HybridClock initialized: now 1786335469260796 us; error 48 us; skew 500 ppm
Aug 10 04:17:49 dist-test-slave-g350 krb5kdc[27686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335469, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:17:49.263690 28570 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:17:49.264160 28570 webserver.cc:460] Webserver started at http://127.15.216.67:40543/ using document root <none> and password file <none>
I20260810 04:17:49.264499 28570 fs_manager.cc:359] Metadata directory not provided
I20260810 04:17:49.264577 28570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:17:49.264726 28570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:17:49 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335469, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:17:49.278609 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "ef708962f9504cc09da2b1f3112d021a"
format_stamp: "Formatted at 2026-08-10 04:17:49 on dist-test-slave-g350"
server_key: "b0e481576bdaee122a4703ef7a9100a0"
server_key_iv: "2cbff7e52252314bdfca9bb65c5c4256"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:49.279150 28570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "ef708962f9504cc09da2b1f3112d021a"
format_stamp: "Formatted at 2026-08-10 04:17:49 on dist-test-slave-g350"
server_key: "b0e481576bdaee122a4703ef7a9100a0"
server_key_iv: "2cbff7e52252314bdfca9bb65c5c4256"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:49.294155 28570 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.002s sys 0.001s
I20260810 04:17:49.321441 28587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:49.321770 28570 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260810 04:17:49.321897 28570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "ef708962f9504cc09da2b1f3112d021a"
format_stamp: "Formatted at 2026-08-10 04:17:49 on dist-test-slave-g350"
server_key: "b0e481576bdaee122a4703ef7a9100a0"
server_key_iv: "2cbff7e52252314bdfca9bb65c5c4256"
server_key_version: "kuduclusterkey@0"
I20260810 04:17:49.322008 28570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:17:49.340915 28570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:17:49.343668 28570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:17:49.343868 28570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:17:49.344192 28570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:17:49.344589 28570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:17:49.344640 28570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:49.344671 28570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:17:49.344691 28570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:17:49.353000 28570 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:43431
I20260810 04:17:49.353505 28570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:17:49.355451 28710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:43431 every 8 connection(s)
I20260810 04:17:49.357025 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 28570
I20260810 04:17:49.357146 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:17:49.357352 16225 external_mini_cluster.cc:1428] Setting key 9aceab7d41f0c438006d29c550bb2a8a
Aug 10 04:17:49 dist-test-slave-g350 krb5kdc[27686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335469, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:17:49.371357 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.355474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:49385 (local address 127.15.216.126:45125)
0810 04:17:49.355525 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.355528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.356945 (+ 1417us) server_negotiation.cc:407] Connection header received
0810 04:17:49.357812 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.357814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.357828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.357861 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:49.358854 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.359606 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.364061 (+ 4455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.364287 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.364384 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.367238 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:49.367239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:49.367240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:49.367251 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:49.369319 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:49.369978 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:49.369980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:49.369981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:49.370050 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:49.370401 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:49.370402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:49.370402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:49.370553 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:49.370613 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.371032 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.371127 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:17:49.372012 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.355140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45125 (local address 127.15.216.67:49385)
0810 04:17:49.356751 (+ 1611us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.356813 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:49.357664 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:49.358078 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:49.358083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:49.358385 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:17:49.358712 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.358720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.362481 (+ 3761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.362482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.363921 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.363926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.364395 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.364397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.364551 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.365186 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:17:49.365210 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:17:49.367083 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:17:49.369437 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:49.369448 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:49.369451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:49.369871 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:49.370128 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:17:49.370129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:17:49.370129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:17:49.370309 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:17:49.370655 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:17:49.370662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:17:49.370860 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 04:17:49.371543 (+ 683us) client_negotiation.cc:240] Negotiation successful
0810 04:17:49.371743 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1551,"thread_start_us":103,"threads_started":1}
I20260810 04:17:49.372503 28711 heartbeater.cc:347] Connected to a master server at 127.15.216.126:45125
I20260810 04:17:49.372623 28711 heartbeater.cc:464] Registering TS with master...
I20260810 04:17:49.372871 28711 heartbeater.cc:511] Master 127.15.216.126:45125 requested a full tablet report, sending...
I20260810 04:17:49.373313 28190 ts_manager.cc:194] Registered new tserver with Master: ef708962f9504cc09da2b1f3112d021a (127.15.216.67:43431)
I20260810 04:17:49.373862 28190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:49385
I20260810 04:17:49.380620 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:17:49.390542 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.382680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.15.216.126:45125)
0810 04:17:49.382777 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.382780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.383005 (+ 225us) server_negotiation.cc:407] Connection header received
0810 04:17:49.384074 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.384076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.384089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.384173 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:17:49.384807 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.385321 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.386154 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.386353 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.386447 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.387816 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:17:49.387817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:17:49.387818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:17:49.387826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:17:49.389300 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:49.389695 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:49.389696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:49.389697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:49.389762 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:17:49.389992 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:17:49.389993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:17:49.389993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:17:49.390119 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:17:49.390230 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.390340 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.390427 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 04:17:49.392524 28190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:17:49.394483 28190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:17:49.402029 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.397426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:32867 (local address 127.0.0.1:41078)
0810 04:17:49.397619 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.397664 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:49.397732 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:49.398310 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:49.398314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:49.398317 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:49.398462 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.398465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.399725 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.399726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.400776 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.400779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.401671 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.401672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.401768 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.401786 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 04:17:49.401869 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 04:17:49.401935 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260810 04:17:49.402034 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.397826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.15.216.65:32867)
0810 04:17:49.397997 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.398002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.398013 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:17:49.398049 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.398051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.398067 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.398169 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:49.398579 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.399642 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.400878 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.401548 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.401668 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.401736 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.401904 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.401948 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260810 04:17:49.402854 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.398009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.15.216.66:39235)
0810 04:17:49.398387 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.398392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.398402 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:17:49.398443 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.398445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.398460 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.398537 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:49.398648 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.399892 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.401211 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.402003 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.402076 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.402153 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.402649 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.402702 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":89,"threads_started":1}
I20260810 04:17:49.402539 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.397469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:39235 (local address 127.0.0.1:45286)
0810 04:17:49.398194 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.398206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:49.398268 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:49.398534 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:49.398535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:49.398537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:49.398587 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.398588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.400001 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.400002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.401087 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.401091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.402190 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.402191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.402265 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.402269 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:17:49.402401 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 04:17:49.402449 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":140,"threads_started":1}
I20260810 04:17:49.403450 28327 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9443d13e4b4209ba9973476e07905e (DEFAULT_TABLE table=test-table [id=db04173457174e22940152e27668adea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:17:49.403795 28327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9443d13e4b4209ba9973476e07905e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:17:49.407130 28730 tablet_bootstrap.cc:492] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7: Bootstrap starting.
I20260810 04:17:49.408048 28730 tablet_bootstrap.cc:654] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7: Neither blocks nor log segments found. Creating new log.
I20260810 04:17:49.408241 28492 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9443d13e4b4209ba9973476e07905e (DEFAULT_TABLE table=test-table [id=db04173457174e22940152e27668adea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:17:49.408511 28730 log.cc:824] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:17:49.408670 28492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9443d13e4b4209ba9973476e07905e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:17:49.409730 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.397749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:43431 (local address 127.0.0.1:55534)
0810 04:17:49.402820 (+ 5071us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.402832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:49.402867 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:49.405945 (+ 3078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:49.405947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:49.405949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:49.405997 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.405999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.407313 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.407314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.408377 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.408382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.409437 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.409438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.409510 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.409516 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:17:49.409548 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 04:17:49.409647 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5057,"thread_start_us":69,"threads_started":1}
I20260810 04:17:49.409930 28730 tablet_bootstrap.cc:492] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7: No bootstrap required, opened a new log
I20260810 04:17:49.410017 28730 ts_tablet_manager.cc:1397] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:17:49.410101 28729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.398012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.15.216.67:43431)
0810 04:17:49.405748 (+ 7736us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.405755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.405769 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:17:49.405817 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.405818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.405836 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.405951 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:49.406105 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.407214 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.408527 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.409314 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.409853 (+ 539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.409947 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.409977 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.410024 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7679,"thread_start_us":85,"threads_started":1}
I20260810 04:17:49.411909 28732 tablet_bootstrap.cc:492] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Bootstrap starting.
I20260810 04:17:49.413019 28732 tablet_bootstrap.cc:654] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Neither blocks nor log segments found. Creating new log.
I20260810 04:17:49.413414 28732 log.cc:824] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Log is configured to *not* fsync() on all Append() calls
I20260810 04:17:49.413311 28637 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9443d13e4b4209ba9973476e07905e (DEFAULT_TABLE table=test-table [id=db04173457174e22940152e27668adea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:17:49.413718 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c9443d13e4b4209ba9973476e07905e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:17:49.414188 28730 raft_consensus.cc:348] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.414414 28730 raft_consensus.cc:374] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:17:49.414426 28732 tablet_bootstrap.cc:492] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: No bootstrap required, opened a new log
I20260810 04:17:49.414458 28730 raft_consensus.cc:729] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ac0ca21ca446f0bc858f981fb918d7, State: Initialized, Role: FOLLOWER
I20260810 04:17:49.414490 28732 ts_tablet_manager.cc:1397] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:17:49.414638 28730 consensus_queue.cc:260] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.414978 28730 ts_tablet_manager.cc:1428] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260810 04:17:49.415120 28416 heartbeater.cc:503] Master 127.15.216.126:45125 was elected leader, sending a full tablet report...
I20260810 04:17:49.416599 28732 raft_consensus.cc:348] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.416769 28732 raft_consensus.cc:374] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:17:49.416805 28732 raft_consensus.cc:729] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c86646d4ebc4fda915098d4ceca3c25, State: Initialized, Role: FOLLOWER
I20260810 04:17:49.416941 28732 consensus_queue.cc:260] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.417228 28732 ts_tablet_manager.cc:1428] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:17:49.417310 28565 heartbeater.cc:503] Master 127.15.216.126:45125 was elected leader, sending a full tablet report...
I20260810 04:17:49.418445 28737 tablet_bootstrap.cc:492] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Bootstrap starting.
I20260810 04:17:49.419425 28737 tablet_bootstrap.cc:654] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Neither blocks nor log segments found. Creating new log.
I20260810 04:17:49.419762 28737 log.cc:824] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:17:49.420598 28737 tablet_bootstrap.cc:492] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: No bootstrap required, opened a new log
I20260810 04:17:49.420742 28737 ts_tablet_manager.cc:1397] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:17:49.422858 28737 raft_consensus.cc:348] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.423110 28737 raft_consensus.cc:374] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:17:49.423188 28737 raft_consensus.cc:729] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef708962f9504cc09da2b1f3112d021a, State: Initialized, Role: FOLLOWER
I20260810 04:17:49.423375 28737 consensus_queue.cc:260] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.423717 28737 ts_tablet_manager.cc:1428] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:17:49.423874 28711 heartbeater.cc:503] Master 127.15.216.126:45125 was elected leader, sending a full tablet report...
I20260810 04:17:49.425323 28735 raft_consensus.cc:482] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:17:49.425457 28735 raft_consensus.cc:504] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.425894 28735 leader_election.cc:288] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ac0ca21ca446f0bc858f981fb918d7 (127.15.216.65:32867), ef708962f9504cc09da2b1f3112d021a (127.15.216.67:43431)
I20260810 04:17:49.429560 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.426149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:43431 (local address 127.15.216.66:57567)
0810 04:17:49.426376 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.426389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:49.426443 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:49.426656 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:49.426657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:49.426659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:49.426731 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.426733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.427705 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.427706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.428536 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.428539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.429318 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.429319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.429393 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.429396 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:17:49.429440 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:17:49.429482 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":25,"thread_start_us":125,"threads_started":1}
I20260810 04:17:49.429855 28729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.426227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:57567 (local address 127.15.216.67:43431)
0810 04:17:49.426282 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.426284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.426503 (+ 219us) server_negotiation.cc:407] Connection header received
0810 04:17:49.426530 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.426531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.426540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.426588 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:49.426811 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.427574 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.428662 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.429206 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.429276 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.429284 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.429760 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.429798 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:17:49.430106 28649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9443d13e4b4209ba9973476e07905e" candidate_uuid: "3c86646d4ebc4fda915098d4ceca3c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef708962f9504cc09da2b1f3112d021a" is_pre_election: true
I20260810 04:17:49.430255 28649 raft_consensus.cc:2454] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c86646d4ebc4fda915098d4ceca3c25 in term 0.
I20260810 04:17:49.430516 28444 leader_election.cc:302] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c86646d4ebc4fda915098d4ceca3c25, ef708962f9504cc09da2b1f3112d021a; no voters:
I20260810 04:17:49.430642 28735 raft_consensus.cc:2781] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:17:49.430716 28735 raft_consensus.cc:482] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:17:49.430740 28735 raft_consensus.cc:3037] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:17:49.431509 28735 raft_consensus.cc:504] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.431644 28735 leader_election.cc:288] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [CANDIDATE]: Term 1 election: Requested vote from peers e2ac0ca21ca446f0bc858f981fb918d7 (127.15.216.65:32867), ef708962f9504cc09da2b1f3112d021a (127.15.216.67:43431)
I20260810 04:17:49.431859 28649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9443d13e4b4209ba9973476e07905e" candidate_uuid: "3c86646d4ebc4fda915098d4ceca3c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef708962f9504cc09da2b1f3112d021a"
I20260810 04:17:49.431946 28649 raft_consensus.cc:3037] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:17:49.432600 28649 raft_consensus.cc:2454] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c86646d4ebc4fda915098d4ceca3c25 in term 1.
I20260810 04:17:49.432776 28444 leader_election.cc:302] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c86646d4ebc4fda915098d4ceca3c25, ef708962f9504cc09da2b1f3112d021a; no voters:
I20260810 04:17:49.432858 28735 raft_consensus.cc:2781] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:17:49.433048 28735 raft_consensus.cc:686] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 1 LEADER]: Becoming Leader. State: Replica: 3c86646d4ebc4fda915098d4ceca3c25, State: Running, Role: LEADER
I20260810 04:17:49.433140 28735 consensus_queue.cc:237] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } }
I20260810 04:17:49.434000 28189 catalog_manager.cc:5495] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c86646d4ebc4fda915098d4ceca3c25 (127.15.216.66). New cstate: current_term: 1 leader_uuid: "3c86646d4ebc4fda915098d4ceca3c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c86646d4ebc4fda915098d4ceca3c25" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 39235 } health_report { overall_health: HEALTHY } } }
I20260810 04:17:49.435642 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.426045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:32867 (local address 127.15.216.66:51839)
0810 04:17:49.426450 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.426459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:17:49.426508 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:17:49.426690 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:17:49.426691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:17:49.426693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:17:49.426771 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.426772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.427715 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.427715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.429895 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:17:49.429899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.435377 (+ 5478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.435378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:17:49.435456 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.435461 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:17:49.435507 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 04:17:49.435552 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":40}
I20260810 04:17:49.435644 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.426386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:51839 (local address 127.15.216.65:32867)
0810 04:17:49.426434 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.426436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.426558 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:17:49.426584 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.426585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.426594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.426640 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:17:49.426846 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.427622 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.430555 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.435268 (+ 4713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.435376 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.435386 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.435526 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.435568 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:17:49.436188 28369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9443d13e4b4209ba9973476e07905e" candidate_uuid: "3c86646d4ebc4fda915098d4ceca3c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" is_pre_election: true
I20260810 04:17:49.436345 28369 raft_consensus.cc:2454] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c86646d4ebc4fda915098d4ceca3c25 in term 0.
I20260810 04:17:49.436477 28368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c9443d13e4b4209ba9973476e07905e" candidate_uuid: "3c86646d4ebc4fda915098d4ceca3c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ac0ca21ca446f0bc858f981fb918d7"
I20260810 04:17:49.436548 28368 raft_consensus.cc:3037] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:17:49.437228 28368 raft_consensus.cc:2454] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c86646d4ebc4fda915098d4ceca3c25 in term 1.
I20260810 04:17:49.448693 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.445414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.15.216.66:39235)
0810 04:17:49.445458 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:17:49.445460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:17:49.445471 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:17:49.445538 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:17:49.445539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:17:49.445563 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:17:49.445669 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:17:49.445798 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.446551 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.447720 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:17:49.448013 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:17:49.448099 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:17:49.448306 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:17:49.448438 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:17:49.448536 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:17:49.448587 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 04:17:49.448613 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:17:49.450995 28649 raft_consensus.cc:1261] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 1 FOLLOWER]: Refusing update from remote peer 3c86646d4ebc4fda915098d4ceca3c25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:17:49.450994 28368 raft_consensus.cc:1261] T 8c9443d13e4b4209ba9973476e07905e P e2ac0ca21ca446f0bc858f981fb918d7 [term 1 FOLLOWER]: Refusing update from remote peer 3c86646d4ebc4fda915098d4ceca3c25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:17:49.451347 28735 consensus_queue.cc:1035] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ac0ca21ca446f0bc858f981fb918d7" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:17:49.451475 28741 consensus_queue.cc:1035] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef708962f9504cc09da2b1f3112d021a" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 43431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:17:49.455368 28745 mvcc.cc:205] Tried to move back new op lower bound from 7316830082868047872 to 7316830082798714880. Current Snapshot: MvccSnapshot[applied={T|T < 7316830082868047872}]
I20260810 04:17:49.455827 28746 mvcc.cc:205] Tried to move back new op lower bound from 7316830082868047872 to 7316830082798714880. Current Snapshot: MvccSnapshot[applied={T|T < 7316830082868047872}]
I20260810 04:17:49.457342 28748 mvcc.cc:205] Tried to move back new op lower bound from 7316830082868047872 to 7316830082798714880. Current Snapshot: MvccSnapshot[applied={T|T < 7316830082868047872}]
I20260810 04:17:49.460193 28190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:17:49.460302 28190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:17:49.461166 28190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ec61d8ccb4144c784b852abc61aefc4: Sending DeleteTablet for 3 replicas of tablet 8c9443d13e4b4209ba9973476e07905e
I20260810 04:17:49.461541 28327 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c9443d13e4b4209ba9973476e07905e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:17:49 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:41078
I20260810 04:17:49.461572 28637 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c9443d13e4b4209ba9973476e07905e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:17:49 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:55534
I20260810 04:17:49.461973 28492 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c9443d13e4b4209ba9973476e07905e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:17:49 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:45286
I20260810 04:17:49.462576 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 28280
I20260810 04:17:49.469754 28757 tablet_replica.cc:330] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: stopping tablet replica
I20260810 04:17:49.469949 28757 raft_consensus.cc:2229] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:17:49.470187 28757 raft_consensus.cc:2258] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:17:49.470834 28757 ts_tablet_manager.cc:1905] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:17:49.472829 28756 tablet_replica.cc:330] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: stopping tablet replica
I20260810 04:17:49.473084 28756 raft_consensus.cc:2229] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:17:49.473302 28756 raft_consensus.cc:2258] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:17:49.473543 28757 ts_tablet_manager.cc:1918] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:17:49.474037 28756 ts_tablet_manager.cc:1905] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:17:49.474170 28757 log.cc:1196] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/8c9443d13e4b4209ba9973476e07905e
I20260810 04:17:49.476114 28757 ts_tablet_manager.cc:1939] T 8c9443d13e4b4209ba9973476e07905e P 3c86646d4ebc4fda915098d4ceca3c25: Deleting consensus metadata
I20260810 04:17:49.476703 28177 catalog_manager.cc:4841] TS 3c86646d4ebc4fda915098d4ceca3c25 (127.15.216.66:39235): tablet 8c9443d13e4b4209ba9973476e07905e (table test-table [id=db04173457174e22940152e27668adea]) successfully deleted
I20260810 04:17:49.482283 28756 ts_tablet_manager.cc:1918] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:17:49.482493 28756 log.cc:1196] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/8c9443d13e4b4209ba9973476e07905e
I20260810 04:17:49.483233 28756 ts_tablet_manager.cc:1939] T 8c9443d13e4b4209ba9973476e07905e P ef708962f9504cc09da2b1f3112d021a: Deleting consensus metadata
I20260810 04:17:49.483841 28175 catalog_manager.cc:4841] TS ef708962f9504cc09da2b1f3112d021a (127.15.216.67:43431): tablet 8c9443d13e4b4209ba9973476e07905e (table test-table [id=db04173457174e22940152e27668adea]) successfully deleted
W20260810 04:17:49.484117 28175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:32867 (error 108)
I20260810 04:17:49.485172 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:17:49.485039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:32867 (local address 127.0.0.1:41092)
0810 04:17:49.485091 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:17:49.485137 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:32867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:17:49.485409 28175 catalog_manager.cc:4584] TS e2ac0ca21ca446f0bc858f981fb918d7 (127.15.216.65:32867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c9443d13e4b4209ba9973476e07905e: Network error: Client connection negotiation failed: client connection to 127.15.216.65:32867: connect: Connection refused (error 111)
I20260810 04:17:49.485729 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 28425
I20260810 04:17:49.499827 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 28570
I20260810 04:17:49.506800 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 28158
I20260810 04:17:49.514798 16225 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 04:17:49.860138 16225 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 04:17:49.860258 16225 mini_ranger.cc:66] Stopping Ranger...
I20260810 04:17:50.207090 16225 mini_ranger.cc:68] Stopped Ranger
2026-08-10 04:17:50.207 UTC [27721] LOG: received smart shutdown request
2026-08-10 04:17:50.211 UTC [27721] LOG: background worker "logical replication launcher" (PID 27731) exited with exit code 1
2026-08-10 04:17:50.212 UTC [27726] LOG: shutting down
2026-08-10 04:17:50.417 UTC [27721] LOG: database system is shut down
2026-08-10T04:17:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50026 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:17:50 dist-test-slave-g350 krb5kdc[28767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:17:50 dist-test-slave-g350 krb5kdc[28767](info): set up 2 sockets
Aug 10 04:17:50 dist-test-slave-g350 krb5kdc[28767](info): commencing operation
krb5kdc: starting...
W20260810 04:17:51.623327 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:17:51 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:17:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/rangeradmin_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/rangeradmin_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/rangerlookup_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/rangerlookup_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
I20260810 04:17:51.804497 16225 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrlBm0x/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/postgres -l logfile start
2026-08-10 04:17:54.535 UTC [28798] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 04:17:54.535 UTC [28798] LOG: listening on IPv4 address "127.15.216.84", port 58541
2026-08-10 04:17:54.539 UTC [28798] LOG: listening on Unix socket "/tmp/.s.PGSQL.58541"
2026-08-10 04:17:54.562 UTC [28801] LOG: database system was shut down at 2026-08-10 04:17:52 UTC
2026-08-10 04:17:54.566 UTC [28798] LOG: database system is ready to accept connections
I20260810 04:17:55.558331 16225 mini_postgres.cc:96] Postgres bound to 58541
2026-08-10 04:17:55.565 UTC [28809] FATAL: database "slave" does not exist
127.15.216.84:58541 - accepting connections
I20260810 04:17:55.566114 16225 mini_ranger.cc:161] Starting Ranger...
I20260810 04:17:55.582399 16225 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 04:17:56.122264 16225 mini_ranger.cc:87] Created ranger Postgres database
I20260810 04:17:56.122357 16225 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-admin
2026-08-10 04:17:56,422 [I] DB FLAVOR :POSTGRES
2026-08-10 04:17:56,423 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:17:56,423 [I] Checking connection..
2026-08-10 04:17:56,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 04:17:56,720 [I] Checking connection passed.
2026-08-10 04:17:56,721 [I] --------- Verifying version history table ---------
2026-08-10 04:17:56,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:17:57,000 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:17:57,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:17:57,270 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 04:17:57,271 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 04:17:57,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 04:17:57.535 UTC [28878] WARNING: there is no transaction in progress
2026-08-10 04:17:57,544 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 04:17:57,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 04:17:57,823 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 04:17:57,824 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:17:57,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 04:17:58,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 04:17:58,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g350.c.gcp-upstream.internal','N') ;"
2026-08-10 04:17:58,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:17:58,940 [I] Table x_portal_user does not exist in database ranger
2026-08-10 04:17:58,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:17:59,215 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 04:17:59,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:17:59,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:17:59,752 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 04:17:59,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 04:18:00.820 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:00.838 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:00.856 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:00.874 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:00.895 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:00.918 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:00.984 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:00.992 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.009 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.022 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.033 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.050 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.061 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.068 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.075 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01.084 UTC [28995] WARNING: there is no transaction in progress
2026-08-10 04:18:01,879 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 04:18:01,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 04:18:02,131 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 04:18:02,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 04:18:02,400 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 04:18:02,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 04:18:02,649 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 04:18:02,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 04:18:02,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 04:18:03,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g350.c.gcp-upstream.internal';"
2026-08-10 04:18:03,484 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 04:18:03,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 04:18:03,758 [I] DB_PATCHES have already been applied
I20260810 04:18:03.764916 16225 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrlBm0x/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 04:18:03.764990 16225 mini_ranger.cc:193] Using host: 127.15.216.84
I20260810 04:18:03.768939 16225 mini_ranger.cc:227] Ranger admin URL: http://127.15.216.84:56189
Aug 10, 2026 4:18:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:18:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:18:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 4:18:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 4:18:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 4:18:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 4:18:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 04:18:04.943082 16225 mini_ranger.cc:160] Time spent starting Ranger: real 9.377s user 0.000s sys 0.011s
Aug 10, 2026 4:18:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 4:18:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 4:18:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 4:18:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 4:18:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 04:18:27.084933 16225 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.216.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.216.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/rangerkms_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.216.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/rangerkms_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.216.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.216.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.216.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/HTTP_127.15.216.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 04:18:27 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335507, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 04:18:27.184352 16225 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 04:18:27.196049 16225 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 04:18:28.220561 16225 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 04:18:28.220649 16225 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms
I20260810 04:18:28.220662 16225 mini_ranger_kms.cc:227] Using postgres at 127.15.216.84:58541
2026-08-10 04:18:28,390 [I] DB FLAVOR :POSTGRES
2026-08-10 04:18:28,390 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 04:18:28,390 [I] Checking connection
2026-08-10 04:18:28,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 04:18:28,648 [I] connection success
2026-08-10 04:18:28,649 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 04:18:28,649 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 04:18:28,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 04:18:28,895 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 04:18:28,895 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 04:18:28,895 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 04:18:28,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.216.84:58541/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 04:18:29,170 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 04:18:29.669778 16225 mini_ranger_kms.cc:310] Created kms service
I20260810 04:18:29.841782 16225 mini_ranger_kms.cc:326] Created kudu user
I20260810 04:18:29.889487 16225 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 04:18:29 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335507, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:30.316903 16225 mini_ranger_kms.cc:384] Added ranger policy
I20260810 04:18:30.317116 16225 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrlBm0x/build/release/bin/postgresql.jar:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrlBm0x/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 04:18:30.317142 16225 mini_ranger_kms.cc:240] Using host: 127.15.216.84
I20260810 04:18:30.322064 16225 mini_ranger_kms.cc:276] Ranger KMS PID: 29192
I20260810 04:18:30.322136 16225 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.216.84:51007
Aug 10, 2026 4:18:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 4:18:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 4:18:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 4:18:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 4:18:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 4:18:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 04:18:32.719359 16225 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.535s user 0.008s sys 0.146s
I20260810 04:18:32.719481 16225 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 04:18:32.719542 16225 mini_ranger_kms.cc:401] 127.15.216.84:51007/kms/v1/keys
Aug 10 04:18:32 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786335512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.216.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 04:18:33 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.216.84@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
Aug 10 04:18:33 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335512, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.216.84@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
Aug 10 04:18:35 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:18:35.912914 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:43847
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:43847
--ranger_config_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:36.019392 29245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:36.019649 29245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:36.019687 29245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:36.020838 29245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:36.020915 29245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:36.020941 29245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:36.020970 29245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:36.021000 29245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:36.022564 29245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.216.84:51007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:43847
--ranger_config_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:43847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:36.022841 29245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:36.023077 29245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:36.028218 29251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:36.028223 29250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:36.028218 29253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:36.028884 29245 server_base.cc:1034] running on GCE node
I20260810 04:18:36.029182 29245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:36.029510 29245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:36.030709 29245 hybrid_clock.cc:648] HybridClock initialized: now 1786335516030679 us; error 46 us; skew 500 ppm
Aug 10 04:18:36 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:36.033732 29245 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:18:36.034209 29245 webserver.cc:460] Webserver started at http://127.15.216.126:35935/ using document root <none> and password file <none>
I20260810 04:18:36.034463 29245 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:36.034531 29245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:36.034703 29245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:18:36 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:36.087733 29245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "419315322015439ba074a21a52261923"
format_stamp: "Formatted at 2026-08-10 04:18:36 on dist-test-slave-g350"
server_key: "d14c27cab3dc8afe9cdda9dc930e1495"
server_key_iv: "60668752fe603bb3717bcac16eb579b8"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:36.088199 29245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "419315322015439ba074a21a52261923"
format_stamp: "Formatted at 2026-08-10 04:18:36 on dist-test-slave-g350"
server_key: "d14c27cab3dc8afe9cdda9dc930e1495"
server_key_iv: "60668752fe603bb3717bcac16eb579b8"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:36.089910 29245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:18:36.103327 29261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:36.103586 29245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:36.103705 29245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "419315322015439ba074a21a52261923"
format_stamp: "Formatted at 2026-08-10 04:18:36 on dist-test-slave-g350"
server_key: "d14c27cab3dc8afe9cdda9dc930e1495"
server_key_iv: "60668752fe603bb3717bcac16eb579b8"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:36.103813 29245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:36.129326 29245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:36.131726 29245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:36.131906 29245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:36.137514 29245 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:43847
I20260810 04:18:36.137614 29319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:43847 every 8 connection(s)
I20260810 04:18:36.138031 29245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:18:36.138833 29320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:36.141336 29320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Bootstrap starting.
I20260810 04:18:36.141669 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 29245
I20260810 04:18:36.141767 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:18:36.141957 16225 external_mini_cluster.cc:1428] Setting key fb660de099f6a0d4b6f783f6b9243ebf
I20260810 04:18:36.142148 29320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:36.142596 29320 log.cc:824] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:36.143591 29320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: No bootstrap required, opened a new log
I20260810 04:18:36.145418 29320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:36.145599 29320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:36.145706 29320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419315322015439ba074a21a52261923, State: Initialized, Role: FOLLOWER
I20260810 04:18:36.145866 29320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:36.145980 29320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:18:36.146090 29320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:18:36.146147 29320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:36.146963 29320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:36.147161 29320 leader_election.cc:302] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419315322015439ba074a21a52261923; no voters:
I20260810 04:18:36.147337 29320 leader_election.cc:288] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:18:36.147378 29325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:18:36.147622 29325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 LEADER]: Becoming Leader. State: Replica: 419315322015439ba074a21a52261923, State: Running, Role: LEADER
I20260810 04:18:36.147716 29320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:36.147753 29325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:36.148175 29327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 419315322015439ba074a21a52261923. Latest consensus state: current_term: 1 leader_uuid: "419315322015439ba074a21a52261923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } } }
I20260810 04:18:36.148376 29327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:36.148319 29328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "419315322015439ba074a21a52261923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } } }
I20260810 04:18:36.148615 29328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:36.148754 29330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:18:36.149283 29330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:18:36.149842 29320 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 10 04:18:36 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:36.159686 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:36.142528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.15.216.126:43847)
0810 04:18:36.142691 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:36.142700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:36.142736 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:18:36.143313 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:36.143322 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:36.143384 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:36.143536 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:36.144431 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:36.146293 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:36.147488 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:36.148184 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:36.148324 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:36.152598 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:36.152693 (+ 95us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:36.152694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:36.152828 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:36.156808 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:36.157343 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:36.157352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:36.157353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:36.157499 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:36.157794 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:36.157795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:36.157796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:36.158281 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:36.159046 (+ 765us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:36.159149 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 04:18:36.159395 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260810 04:18:36.162019 29330 catalog_manager.cc:1347] Generated new cluster ID: c7814a99a4a448328b655e7330165c5c
I20260810 04:18:36.162077 29330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:18:36.182291 29330 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:18:36.183188 29330 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:18:36.189965 29330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Generated new TSK 0
I20260810 04:18:36.190163 29330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:18:36.926992 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:43847
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:37.043347 29366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:37.043617 29366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:37.043653 29366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:37.045127 29366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:37.045192 29366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:37.045244 29366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:18:37.046885 29366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.216.84:51007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:37.047165 29366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:37.047434 29366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:37.054031 29374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:37.054356 29366 server_base.cc:1034] running on GCE node
W20260810 04:18:37.054396 29371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.054443 29372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:37.054678 29366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:37.054947 29366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:37.056118 29366 hybrid_clock.cc:648] HybridClock initialized: now 1786335517056094 us; error 38 us; skew 500 ppm
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:37.059247 29366 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:18:37.059723 29366 webserver.cc:460] Webserver started at http://127.15.216.65:40277/ using document root <none> and password file <none>
I20260810 04:18:37.060001 29366 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:37.060081 29366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:37.060249 29366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:37.077391 29366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "c9abbbd3e47c4440b122f7e356a92eba"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "823e33fb2d11753a646bd68e7161abd1"
server_key_iv: "3066ef99e911a75960bd8ced33be5378"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.078047 29366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "c9abbbd3e47c4440b122f7e356a92eba"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "823e33fb2d11753a646bd68e7161abd1"
server_key_iv: "3066ef99e911a75960bd8ced33be5378"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.083195 29366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260810 04:18:37.102099 29386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.102289 29366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 04:18:37.102392 29366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "c9abbbd3e47c4440b122f7e356a92eba"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "823e33fb2d11753a646bd68e7161abd1"
server_key_iv: "3066ef99e911a75960bd8ced33be5378"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.102509 29366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:37.126448 29366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:37.128929 29366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:37.129107 29366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:37.129364 29366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:37.129796 29366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:37.129838 29366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.129882 29366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:37.129908 29366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.137006 29366 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:35373
I20260810 04:18:37.137107 29506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:35373 every 8 connection(s)
I20260810 04:18:37.137456 29366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:37.145565 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 29366
I20260810 04:18:37.145707 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:18:37.145926 16225 external_mini_cluster.cc:1428] Setting key a81419d1073b5f104e41fca45b4b81fb
I20260810 04:18:37.147167 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:37.138217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:55455 (local address 127.15.216.126:43847)
0810 04:18:37.138356 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:37.138359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:37.138552 (+ 193us) server_negotiation.cc:407] Connection header received
0810 04:18:37.139044 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:37.139045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:37.139058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:37.139125 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:37.139665 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.140173 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.141142 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.141365 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.141451 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:37.143905 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:37.143907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:37.143907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:37.143915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:37.145389 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:37.145865 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:37.145866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:37.145867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:37.145930 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:37.146229 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:37.146230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:37.146231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:37.146374 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:37.146440 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:37.146940 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 04:18:37.147033 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260810 04:18:37.147344 29509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:37.138198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:43847 (local address 127.15.216.65:55455)
0810 04:18:37.138418 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:37.138458 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:37.138943 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:37.139190 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:37.139194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:37.139392 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:37.139574 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:37.139578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.140272 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.140273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:37.141050 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:37.141052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.141476 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.141478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:37.141633 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:37.142228 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:37.142236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:37.143812 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:37.145484 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:37.145490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:37.145492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:37.145789 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:37.145998 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:37.145999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:37.145999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:37.146148 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:37.146467 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:37.146475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:37.146769 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 04:18:37.146904 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 04:18:37.147076 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
I20260810 04:18:37.148054 29507 heartbeater.cc:347] Connected to a master server at 127.15.216.126:43847
I20260810 04:18:37.148160 29507 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:37.148394 29507 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:37.149015 29278 ts_manager.cc:194] Registered new tserver with Master: c9abbbd3e47c4440b122f7e356a92eba (127.15.216.65:35373)
I20260810 04:18:37.150164 29278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:55455
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:18:37.178201 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:43847
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:37.277289 29513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:37.277568 29513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:37.277631 29513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:37.278864 29513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:37.278934 29513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:37.278978 29513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:18:37.280624 29513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.216.84:51007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:37.280905 29513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:37.281148 29513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:37.286576 29522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.286689 29521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 04:18:37.286583 29524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:37.287285 29513 server_base.cc:1034] running on GCE node
I20260810 04:18:37.287511 29513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:37.287773 29513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:37.288935 29513 hybrid_clock.cc:648] HybridClock initialized: now 1786335517288910 us; error 36 us; skew 500 ppm
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:37.291440 29513 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:18:37.291947 29513 webserver.cc:460] Webserver started at http://127.15.216.66:36823/ using document root <none> and password file <none>
I20260810 04:18:37.292192 29513 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:37.292260 29513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:37.292414 29513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:37.305768 29513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "c6205d7092dc40069ed0824007d26731"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "d31d99a83361fe1fb9a12ff2557a5626"
server_key_iv: "b9a2067d46691a8e5f187b5a9e96bcf9"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.306221 29513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "c6205d7092dc40069ed0824007d26731"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "d31d99a83361fe1fb9a12ff2557a5626"
server_key_iv: "b9a2067d46691a8e5f187b5a9e96bcf9"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.317951 29513 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.000s
I20260810 04:18:37.331801 29532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.332041 29513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:37.332124 29513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "c6205d7092dc40069ed0824007d26731"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "d31d99a83361fe1fb9a12ff2557a5626"
server_key_iv: "b9a2067d46691a8e5f187b5a9e96bcf9"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.332211 29513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:37.347494 29513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:37.349982 29513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:37.350149 29513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:37.350402 29513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:37.350783 29513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:37.350834 29513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.350867 29513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:37.350891 29513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.358366 29513 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:45741
I20260810 04:18:37.358819 29513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:18:37.359781 29654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:45741 every 8 connection(s)
I20260810 04:18:37.366653 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 29513
I20260810 04:18:37.366768 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:37.366992 16225 external_mini_cluster.cc:1428] Setting key f937b382194bd435938b05d87f507c0c
I20260810 04:18:37.371012 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:37.360488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:36495 (local address 127.15.216.126:43847)
0810 04:18:37.360543 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:37.360545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:37.360854 (+ 309us) server_negotiation.cc:407] Connection header received
0810 04:18:37.361490 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:37.361492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:37.361508 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:37.361630 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:37.362328 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.363094 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.364550 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.364852 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.364901 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:37.367522 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:37.367524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:37.367524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:37.367533 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:37.369320 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:37.369849 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:37.369850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:37.369851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:37.369914 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:37.370227 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:37.370229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:37.370229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:37.370387 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:37.370444 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:37.370818 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 04:18:37.370914 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:18:37.371531 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:37.360457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:43847 (local address 127.15.216.66:36495)
0810 04:18:37.360705 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:37.360746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:37.361375 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:37.361692 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:37.361696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:37.361945 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:37.362212 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:37.362218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.363212 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.363213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:37.364439 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:37.364442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.365014 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.365015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:37.365166 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:37.365823 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:37.365834 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:37.367293 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:37.369415 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:37.369421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:37.369424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:37.369748 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:37.369994 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:37.369994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:37.369994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:37.370137 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:37.370483 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:37.370489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:37.370649 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 04:18:37.371119 (+ 470us) client_negotiation.cc:240] Negotiation successful
0810 04:18:37.371298 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260810 04:18:37.372176 29655 heartbeater.cc:347] Connected to a master server at 127.15.216.126:43847
I20260810 04:18:37.372290 29655 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:37.372485 29655 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:37.372839 29278 ts_manager.cc:194] Registered new tserver with Master: c6205d7092dc40069ed0824007d26731 (127.15.216.66:45741)
I20260810 04:18:37.373287 29278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:36495
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:18:37.401914 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:43847
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:37.515136 29661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:37.515414 29661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:37.515453 29661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:37.517143 29661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:37.517199 29661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:37.517238 29661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:18:37.519388 29661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.216.84:51007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:37.519635 29661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:37.519904 29661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:37.526369 29666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.526364 29670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.527022 29667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:37.527268 29661 server_base.cc:1034] running on GCE node
I20260810 04:18:37.527498 29661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:37.527758 29661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:37.528942 29661 hybrid_clock.cc:648] HybridClock initialized: now 1786335517528919 us; error 36 us; skew 500 ppm
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:37.532183 29661 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:18:37.532727 29661 webserver.cc:460] Webserver started at http://127.15.216.67:40525/ using document root <none> and password file <none>
I20260810 04:18:37.533010 29661 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:37.533071 29661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:37.533205 29661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:37.546512 29661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "536ebb41ecde40e4ae298c1f11184120"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "a633295723fece3594e404906e1ee527"
server_key_iv: "bcb1864b3abf197299e5425b6e7b5ad7"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.546944 29661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "536ebb41ecde40e4ae298c1f11184120"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "a633295723fece3594e404906e1ee527"
server_key_iv: "bcb1864b3abf197299e5425b6e7b5ad7"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.550005 29661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 04:18:37.562511 29677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.562763 29661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:18:37.562871 29661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "536ebb41ecde40e4ae298c1f11184120"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "a633295723fece3594e404906e1ee527"
server_key_iv: "bcb1864b3abf197299e5425b6e7b5ad7"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.562991 29661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:37.580502 29661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:37.583016 29661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:37.583174 29661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:37.583431 29661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:37.583834 29661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:37.583896 29661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.583935 29661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:37.583961 29661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.590678 29661 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:32953
I20260810 04:18:37.591120 29661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:18:37.594092 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 29661
I20260810 04:18:37.594197 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:18:37.594409 16225 external_mini_cluster.cc:1428] Setting key 8c19037d09d4e41fbece2eba4434cf0d
I20260810 04:18:37.603826 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:32953 every 8 connection(s)
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:37.616680 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:37.604794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:40321 (local address 127.15.216.126:43847)
0810 04:18:37.604840 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:37.604842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:37.605447 (+ 605us) server_negotiation.cc:407] Connection header received
0810 04:18:37.606211 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:37.606212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:37.606226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:37.606259 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:37.607217 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.608032 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.609626 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.609922 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.609973 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:37.612598 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:37.612600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:37.612601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:37.612609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:37.614469 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:37.615488 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:37.615489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:37.615490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:37.615554 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:37.615862 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:37.615863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:37.615863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:37.616028 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:37.616074 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:37.616487 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 04:18:37.616579 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:18:37.617208 29805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:37.604491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:43847 (local address 127.15.216.67:40321)
0810 04:18:37.605310 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:37.605359 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:37.606082 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:37.606445 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:37.606449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:37.606785 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:37.607073 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:37.607079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.608184 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.608184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:37.609489 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:37.609492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:37.610084 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:37.610085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:37.610231 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:37.610862 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:37.610872 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:37.612488 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:37.615058 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:37.615065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:37.615068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:37.615381 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:37.615627 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:37.615628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:37.615628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:37.615780 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:37.616124 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:37.616132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:37.616297 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 04:18:37.616795 (+ 498us) client_negotiation.cc:240] Negotiation successful
0810 04:18:37.616966 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":93,"threads_started":1}
I20260810 04:18:37.617676 29801 heartbeater.cc:347] Connected to a master server at 127.15.216.126:43847
I20260810 04:18:37.617787 29801 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:37.617990 29801 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:37.618355 29278 ts_manager.cc:194] Registered new tserver with Master: 536ebb41ecde40e4ae298c1f11184120 (127.15.216.67:32953)
I20260810 04:18:37.618942 29278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:40321
I20260810 04:18:37.627719 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:18:37.628412 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 29366
I20260810 04:18:37.643957 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 29513
I20260810 04:18:37.652444 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 29661
I20260810 04:18:37.660933 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 29245
I20260810 04:18:37.671344 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:43847
--webserver_interface=127.15.216.126
--webserver_port=35935
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:43847
--ranger_config_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:37.758013 29806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:37.758215 29806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:37.758239 29806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:37.759373 29806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:37.759420 29806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:37.759435 29806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:37.759449 29806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:37.759460 29806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:37.761103 29806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:43847
--ranger_config_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:43847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:37.761353 29806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:37.761623 29806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:37.766755 29814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.766758 29811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.766705 29812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:37.766956 29806 server_base.cc:1034] running on GCE node
I20260810 04:18:37.767371 29806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:37.767637 29806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:37.768805 29806 hybrid_clock.cc:648] HybridClock initialized: now 1786335517768780 us; error 46 us; skew 500 ppm
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:37.771435 29806 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:18:37.771901 29806 webserver.cc:460] Webserver started at http://127.15.216.126:35935/ using document root <none> and password file <none>
I20260810 04:18:37.772136 29806 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:37.772214 29806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:37.816156 29806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:37.817056 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.817333 29806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:37.817453 29806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "419315322015439ba074a21a52261923"
format_stamp: "Formatted at 2026-08-10 04:18:36 on dist-test-slave-g350"
server_key: "d14c27cab3dc8afe9cdda9dc930e1495"
server_key_iv: "60668752fe603bb3717bcac16eb579b8"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.817919 29806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:37.832459 29806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:37.834931 29806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:37.835124 29806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:37.839887 29806 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:43847
I20260810 04:18:37.839941 29877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:43847 every 8 connection(s)
I20260810 04:18:37.840337 29806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:18:37.841523 29878 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:18:37.842228 29878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Bootstrap starting.
I20260810 04:18:37.848024 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 29806
I20260810 04:18:37.848439 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:35373
--local_ip_for_outbound_sockets=127.15.216.65
--tserver_master_addrs=127.15.216.126:43847
--webserver_port=40277
--webserver_interface=127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:37.851101 29878 log.cc:824] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:37.853158 29878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:18:37.853443 29878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Bootstrap complete.
I20260810 04:18:37.855953 29878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:37.856215 29878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419315322015439ba074a21a52261923, State: Initialized, Role: FOLLOWER
I20260810 04:18:37.856412 29878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:37.856495 29878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:18:37.856549 29878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:18:37.856601 29878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:18:37.857647 29878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:37.857776 29878 leader_election.cc:302] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 419315322015439ba074a21a52261923; no voters:
I20260810 04:18:37.857980 29878 leader_election.cc:288] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:18:37.858016 29882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:18:37.858132 29882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [term 2 LEADER]: Becoming Leader. State: Replica: 419315322015439ba074a21a52261923, State: Running, Role: LEADER
I20260810 04:18:37.858229 29882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } }
I20260810 04:18:37.858264 29878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:37.858515 29883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 419315322015439ba074a21a52261923. Latest consensus state: current_term: 2 leader_uuid: "419315322015439ba074a21a52261923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } } }
I20260810 04:18:37.858786 29883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:37.858693 29884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "419315322015439ba074a21a52261923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "419315322015439ba074a21a52261923" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 43847 } } }
I20260810 04:18:37.859076 29884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:37.859094 29892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:18:37.859683 29892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:18:37.859865 29892 catalog_manager.cc:1259] Loaded cluster ID: c7814a99a4a448328b655e7330165c5c
I20260810 04:18:37.859915 29892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:18:37.860044 29878 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 04:18:37.863823 29892 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:18:37.864400 29892 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Loaded TSK: 0
I20260810 04:18:37.864691 29892 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:18:37.940836 29880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:37.941123 29880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:37.941164 29880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:37.942958 29880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:37.943097 29880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:37.943195 29880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:18:37.945662 29880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:35373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=40277
--tserver_master_addrs=127.15.216.126:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:37.946190 29880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:37.946492 29880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:37.952781 29921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.952801 29918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:37.952809 29919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:37.954066 29880 server_base.cc:1034] running on GCE node
I20260810 04:18:37.954285 29880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:37.954566 29880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:37.955754 29880 hybrid_clock.cc:648] HybridClock initialized: now 1786335517955728 us; error 48 us; skew 500 ppm
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:37.959185 29880 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:18:37.959884 29880 webserver.cc:460] Webserver started at http://127.15.216.65:40277/ using document root <none> and password file <none>
I20260810 04:18:37.960170 29880 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:37.960251 29880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:18:37 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:37.991034 29880 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 04:18:37.992100 29931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:37.992309 29880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:37.992400 29880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "c9abbbd3e47c4440b122f7e356a92eba"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "823e33fb2d11753a646bd68e7161abd1"
server_key_iv: "3066ef99e911a75960bd8ced33be5378"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:37.992866 29880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:38.010102 29880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:38.012791 29880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:38.013010 29880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:38.013264 29880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:38.013700 29880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:38.013803 29880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.013870 29880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:38.013917 29880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.020726 29880 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:35373
I20260810 04:18:38.021382 29880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:18:38.026422 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 29880
I20260810 04:18:38.026741 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:45741
--local_ip_for_outbound_sockets=127.15.216.66
--tserver_master_addrs=127.15.216.126:43847
--webserver_port=36823
--webserver_interface=127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:38.030264 30051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:35373 every 8 connection(s)
Aug 10 04:18:38 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:38.048090 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:38.030980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:43847 (local address 127.15.216.65:55641)
0810 04:18:38.032350 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:38.032397 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:38.033157 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:38.033842 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:38.033845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:38.034083 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:38.034341 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:38.034347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.036099 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.036100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:38.037342 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:38.037345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.038338 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.038340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:38.038471 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:38.039146 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:38.039166 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:38.043637 (+ 4471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:38.046137 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:38.046145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:38.046147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:38.046446 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:38.046756 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:38.046757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:38.046757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:38.046883 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:38.047384 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:38.047390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:38.047549 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 04:18:38.047670 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 04:18:38.047844 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"thread_start_us":95,"threads_started":1}
I20260810 04:18:38.048630 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:38.031376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:55641 (local address 127.15.216.126:43847)
0810 04:18:38.032245 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:38.032257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:38.032709 (+ 452us) server_negotiation.cc:407] Connection header received
0810 04:18:38.033457 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:38.033464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:38.033521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:38.033684 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:38.034467 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.035976 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.037443 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.038107 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.038220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:38.043804 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:38.043808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:38.043809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:38.043820 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:38.046036 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.046545 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.046549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.046549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.046669 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.046955 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.046956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.046956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.047264 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:38.047341 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:38.048295 (+ 954us) server_negotiation.cc:299] Negotiation successful
0810 04:18:38.048415 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":80,"threads_started":1}
I20260810 04:18:38.049295 30052 heartbeater.cc:347] Connected to a master server at 127.15.216.126:43847
I20260810 04:18:38.049388 30052 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.049564 30052 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
W20260810 04:18:38.050473 30052 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:43847 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:18:38.050539 30052 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.050594 30052 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:38.050971 30052 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.051044 30052 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
W20260810 04:18:38.051440 30052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:18:38.153375 30054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:38.153741 30054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:38.153813 30054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:38.155475 30054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:38.155604 30054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:38.155673 30054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:18:38.157333 30054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:45741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=36823
--tserver_master_addrs=127.15.216.126:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:38.157680 30054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:38.157933 30054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:38.165407 30064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:38.165513 30066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:38.165833 30054 server_base.cc:1034] running on GCE node
W20260810 04:18:38.165833 30063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:38.166188 30054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:38.166479 30054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:38.167689 30054 hybrid_clock.cc:648] HybridClock initialized: now 1786335518167675 us; error 29 us; skew 500 ppm
Aug 10 04:18:38 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:38.171177 30054 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:18:38.171731 30054 webserver.cc:460] Webserver started at http://127.15.216.66:36823/ using document root <none> and password file <none>
I20260810 04:18:38.172060 30054 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:38.172186 30054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:18:38 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:38.200848 30054 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:38.202045 30074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.202221 30054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:38.202320 30054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "c6205d7092dc40069ed0824007d26731"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "d31d99a83361fe1fb9a12ff2557a5626"
server_key_iv: "b9a2067d46691a8e5f187b5a9e96bcf9"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:38.202708 30054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:38.226430 30054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:38.228657 30054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:38.228853 30054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:38.229092 30054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:38.229450 30054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:38.229509 30054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.229552 30054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:38.229581 30054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.236747 30054 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:45741
I20260810 04:18:38.236822 30196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:45741 every 8 connection(s)
I20260810 04:18:38.237232 30054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:18:38.238354 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30054
I20260810 04:18:38.238668 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.216.84:51007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:32953
--local_ip_for_outbound_sockets=127.15.216.67
--tserver_master_addrs=127.15.216.126:43847
--webserver_port=40525
--webserver_interface=127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:18:38 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:38.252194 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:38.238059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:42777 (local address 127.15.216.126:43847)
0810 04:18:38.238116 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:38.238119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:38.238355 (+ 236us) server_negotiation.cc:407] Connection header received
0810 04:18:38.238995 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:38.238996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:38.239010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:38.239064 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:38.239624 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.240131 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.241723 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.241937 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.241971 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:38.248568 (+ 6597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:38.248571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:38.248571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:38.248581 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:38.250452 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.251085 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.251086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.251087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.251149 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.251422 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.251423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.251423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.251555 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:38.251597 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:38.251996 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 04:18:38.252091 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:18:38.252959 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:38.238028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:43847 (local address 127.15.216.66:42777)
0810 04:18:38.238201 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:38.238243 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:38.238863 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:38.239131 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:38.239136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:38.239331 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:38.239524 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:38.239528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.240229 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.240230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:38.241615 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:38.241618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.245770 (+ 4152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.245771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:38.245958 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:38.246597 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:38.246608 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:38.248459 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:38.250557 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:38.250565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:38.250567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:38.250985 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:38.251224 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:38.251224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:38.251225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:38.251350 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:38.251655 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:38.251661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:38.251833 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 04:18:38.252544 (+ 711us) client_negotiation.cc:240] Negotiation successful
0810 04:18:38.252713 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260810 04:18:38.253737 30197 heartbeater.cc:347] Connected to a master server at 127.15.216.126:43847
I20260810 04:18:38.253839 30197 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.254020 30197 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
W20260810 04:18:38.254654 30197 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:43847 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:18:38.254716 30197 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.254783 30197 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:38.255098 30197 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.255164 30197 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
W20260810 04:18:38.255450 30197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:18:38.340751 30200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:38.340979 30200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:38.341019 30200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:38.342609 30200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:38.342720 30200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:38.342810 30200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:18:38.344959 30200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.216.84:51007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:32953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=40525
--tserver_master_addrs=127.15.216.126:43847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:38.345424 30200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:38.345777 30200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:38.354252 30209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:38.354252 30207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:38.358228 30205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:38.359012 30200 server_base.cc:1034] running on GCE node
I20260810 04:18:38.359333 30200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:38.359658 30200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:38.360828 30200 hybrid_clock.cc:648] HybridClock initialized: now 1786335518360806 us; error 40 us; skew 500 ppm
Aug 10 04:18:38 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:38.363889 30200 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:18:38.364395 30200 webserver.cc:460] Webserver started at http://127.15.216.67:40525/ using document root <none> and password file <none>
I20260810 04:18:38.364632 30200 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:38.364715 30200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 04:18:38 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:38.382795 30200 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:38.383523 30217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.383774 30200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:38.383929 30200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "536ebb41ecde40e4ae298c1f11184120"
format_stamp: "Formatted at 2026-08-10 04:18:37 on dist-test-slave-g350"
server_key: "a633295723fece3594e404906e1ee527"
server_key_iv: "bcb1864b3abf197299e5425b6e7b5ad7"
server_key_version: "kuduclusterkey@0"
I20260810 04:18:38.384482 30200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:38.407833 30200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:38.410703 30200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:38.410983 30200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:38.411332 30200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:38.411803 30200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:38.411900 30200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.411983 30200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:38.412051 30200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:38.419086 30200 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:32953
I20260810 04:18:38.419317 30336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:32953 every 8 connection(s)
I20260810 04:18:38.419584 30200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:18:38.426539 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30200
Aug 10 04:18:38 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:38.431210 30056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:38.420400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:42581 (local address 127.15.216.126:43847)
0810 04:18:38.420479 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:38.420481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:38.420758 (+ 277us) server_negotiation.cc:407] Connection header received
0810 04:18:38.421684 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:38.421690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:38.421701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:38.421748 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:38.422481 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.423168 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.424632 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.424940 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.425022 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:38.427681 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:38.427682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:38.427683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:38.427690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:38.429504 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.430067 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.430068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.430069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.430134 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.430445 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.430446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.430447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.430589 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:38.430652 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:38.431024 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 04:18:38.431113 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:18:38.431396 30339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:38.420380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:43847 (local address 127.15.216.67:42581)
0810 04:18:38.420590 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:38.420641 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:38.421560 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:38.421821 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:38.421825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:38.422092 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:38.422379 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:38.422386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.423288 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.423289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:38.424507 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:38.424511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.425061 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.425062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:38.425205 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:38.425849 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:38.425859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:38.427573 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:38.429636 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:38.429644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:38.429647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:38.429970 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:38.430210 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:38.430211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:38.430212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:38.430362 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:38.430682 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:38.430687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:38.430852 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 04:18:38.430986 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 04:18:38.431149 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260810 04:18:38.431844 30337 heartbeater.cc:347] Connected to a master server at 127.15.216.126:43847
I20260810 04:18:38.431941 30337 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.432092 30337 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
W20260810 04:18:38.432758 30337 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:43847 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 04:18:38.432832 30337 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.432888 30337 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:38.433172 30337 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:38.433249 30337 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
W20260810 04:18:38.433876 30337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:18:38.437959 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:38.427318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.15.216.126:43847)
0810 04:18:38.427717 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:38.427720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:38.428120 (+ 400us) server_negotiation.cc:407] Connection header received
0810 04:18:38.428178 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:38.428179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:38.428193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:38.428272 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:38.430283 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.431009 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.432731 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:38.433044 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:38.433114 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:38.434880 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:38.434882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:38.434882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:38.434890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:38.436574 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.437003 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.437004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.437004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.437063 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:38.437270 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:38.437271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:38.437271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:38.437408 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:38.437584 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:38.437684 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 04:18:38.437779 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":72,"threads_started":1}
Aug 10 04:18:39 dist-test-slave-g350 krb5kdc[28767](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786335519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:39.051630 30052 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:39.051800 30052 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:39.052264 29841 ts_manager.cc:194] Registered new tserver with Master: c9abbbd3e47c4440b122f7e356a92eba (127.15.216.65:35373)
I20260810 04:18:39.053447 29841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:55641
I20260810 04:18:39.255647 30197 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:39.255818 30197 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:39.256234 29841 ts_manager.cc:194] Registered new tserver with Master: c6205d7092dc40069ed0824007d26731 (127.15.216.66:45741)
I20260810 04:18:39.256732 29841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:42777
Aug 10 04:18:39 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:39.434034 30337 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:39.434182 30337 heartbeater.cc:511] Master 127.15.216.126:43847 requested a full tablet report, sending...
I20260810 04:18:39.434502 29841 ts_manager.cc:194] Registered new tserver with Master: 536ebb41ecde40e4ae298c1f11184120 (127.15.216.67:32953)
I20260810 04:18:39.435014 29841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:42581
I20260810 04:18:39.444602 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:18:39.455463 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.445944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.15.216.126:43847)
0810 04:18:39.446330 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:39.446332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:39.446461 (+ 129us) server_negotiation.cc:407] Connection header received
0810 04:18:39.446572 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:39.446574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:39.446589 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:39.446637 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:39.447518 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.448039 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.450090 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.450365 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.450414 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.452224 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:39.452226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:39.452226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:39.452234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:39.454084 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:39.454482 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:39.454483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:39.454484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:39.454590 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:39.454833 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:39.454834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:39.454834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:39.454971 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:39.455052 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:39.455270 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 04:18:39.455360 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":82,"threads_started":1}
I20260810 04:18:39.459161 29841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:18:39.461452 29841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:18:39.470259 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.464502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:35373 (local address 127.0.0.1:48432)
0810 04:18:39.465081 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:39.465089 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:39.465120 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:39.465479 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:39.465482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:39.465484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:39.465557 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.465559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.466917 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.466918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.468567 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.468571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.469951 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.469952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.470039 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.470052 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:18:39.470120 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:18:39.470169 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":89,"threads_started":1}
I20260810 04:18:39.470619 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.464865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.15.216.65:35373)
0810 04:18:39.465221 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:39.465227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:39.465236 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:18:39.465347 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:39.465349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:39.465365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:39.465665 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:39.465737 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.466822 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.468671 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.469454 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.469537 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.469918 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:39.470481 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 04:18:39.470532 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":70,"threads_started":1}
I20260810 04:18:39.471338 29972 tablet_service.cc:1467] Processing CreateTablet for tablet 09bc6a8a30c24d90b32344080cd02799 (DEFAULT_TABLE table=test-table [id=08d21bee494749019798adbf60657b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:39.471738 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bc6a8a30c24d90b32344080cd02799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:39.474143 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.464500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:45741 (local address 127.0.0.1:41894)
0810 04:18:39.464886 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:39.464926 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:39.464987 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:39.467154 (+ 2167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:39.467155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:39.467157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:39.467202 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.467203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.468476 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.468477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.469614 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.469616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.473906 (+ 4290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.473907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.473975 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.473980 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:18:39.474012 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 04:18:39.474056 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":77,"threads_started":1}
I20260810 04:18:39.474684 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.465119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.15.216.66:45741)
0810 04:18:39.466990 (+ 1871us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:39.466995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:39.467005 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:18:39.467039 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:39.467041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:39.467057 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:39.467280 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:39.467348 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.468374 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.469746 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.473801 (+ 4055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.474453 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.474528 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:39.474559 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 04:18:39.474603 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":60,"threads_started":1}
I20260810 04:18:39.475499 30125 tablet_service.cc:1467] Processing CreateTablet for tablet 09bc6a8a30c24d90b32344080cd02799 (DEFAULT_TABLE table=test-table [id=08d21bee494749019798adbf60657b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:39.475858 30125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bc6a8a30c24d90b32344080cd02799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:39.479439 30372 tablet_bootstrap.cc:492] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: Bootstrap starting.
I20260810 04:18:39.480262 30373 tablet_bootstrap.cc:492] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba: Bootstrap starting.
I20260810 04:18:39.480468 30372 tablet_bootstrap.cc:654] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:39.480927 30373 tablet_bootstrap.cc:654] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:39.480911 30372 log.cc:824] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:39.481319 30373 log.cc:824] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:39.481480 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.464722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:32953 (local address 127.0.0.1:54062)
0810 04:18:39.470909 (+ 6187us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:39.470921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:39.470965 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:39.474006 (+ 3041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:39.474007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:39.474009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:39.474041 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.474042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.475274 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.475275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.476309 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.476312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.481242 (+ 4930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.481244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.481317 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.481323 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:18:39.481356 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 04:18:39.481402 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6176,"thread_start_us":73,"threads_started":1}
I20260810 04:18:39.481962 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.465288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.15.216.67:32953)
0810 04:18:39.473758 (+ 8470us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:39.473763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:39.473775 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:18:39.473811 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:39.473813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:39.473829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:39.473918 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:39.474118 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.475167 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.476420 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.477161 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.481696 (+ 4535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.481790 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:39.481823 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 04:18:39.481877 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8421,"thread_start_us":58,"threads_started":1}
I20260810 04:18:39.482704 30268 tablet_service.cc:1467] Processing CreateTablet for tablet 09bc6a8a30c24d90b32344080cd02799 (DEFAULT_TABLE table=test-table [id=08d21bee494749019798adbf60657b92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:39.482981 30372 tablet_bootstrap.cc:492] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: No bootstrap required, opened a new log
I20260810 04:18:39.483047 30372 ts_tablet_manager.cc:1397] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:18:39.483070 30268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bc6a8a30c24d90b32344080cd02799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:39.486258 30373 tablet_bootstrap.cc:492] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba: No bootstrap required, opened a new log
I20260810 04:18:39.486219 30372 raft_consensus.cc:348] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.486413 30372 raft_consensus.cc:374] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:39.486462 30372 raft_consensus.cc:729] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6205d7092dc40069ed0824007d26731, State: Initialized, Role: FOLLOWER
I20260810 04:18:39.486387 30373 ts_tablet_manager.cc:1397] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 04:18:39.486593 30372 consensus_queue.cc:260] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.486900 30372 ts_tablet_manager.cc:1428] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 04:18:39.487028 30197 heartbeater.cc:503] Master 127.15.216.126:43847 was elected leader, sending a full tablet report...
I20260810 04:18:39.487885 30377 tablet_bootstrap.cc:492] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Bootstrap starting.
W20260810 04:18:39.488454 30198 tablet.cc:2369] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:18:39.488552 30377 tablet_bootstrap.cc:654] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:39.488937 30377 log.cc:824] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:39.489800 30377 tablet_bootstrap.cc:492] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: No bootstrap required, opened a new log
I20260810 04:18:39.489867 30377 ts_tablet_manager.cc:1397] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:18:39.490881 30373 raft_consensus.cc:348] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.491036 30373 raft_consensus.cc:374] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:39.491065 30373 raft_consensus.cc:729] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9abbbd3e47c4440b122f7e356a92eba, State: Initialized, Role: FOLLOWER
I20260810 04:18:39.491195 30373 consensus_queue.cc:260] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.491442 30373 ts_tablet_manager.cc:1428] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 04:18:39.491547 30052 heartbeater.cc:503] Master 127.15.216.126:43847 was elected leader, sending a full tablet report...
I20260810 04:18:39.492498 30377 raft_consensus.cc:348] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.492648 30377 raft_consensus.cc:374] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:39.492683 30377 raft_consensus.cc:729] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536ebb41ecde40e4ae298c1f11184120, State: Initialized, Role: FOLLOWER
I20260810 04:18:39.492805 30377 consensus_queue.cc:260] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.493054 30377 ts_tablet_manager.cc:1428] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:18:39.493093 30337 heartbeater.cc:503] Master 127.15.216.126:43847 was elected leader, sending a full tablet report...
W20260810 04:18:39.538831 30053 tablet.cc:2369] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:18:39.630995 30379 raft_consensus.cc:482] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:39.631179 30379 raft_consensus.cc:504] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.631688 30379 leader_election.cc:288] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6205d7092dc40069ed0824007d26731 (127.15.216.66:45741), 536ebb41ecde40e4ae298c1f11184120 (127.15.216.67:32953)
I20260810 04:18:39.636072 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.632152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:53473 (local address 127.15.216.66:45741)
0810 04:18:39.632214 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:39.632217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:39.632350 (+ 133us) server_negotiation.cc:407] Connection header received
0810 04:18:39.632469 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:39.632470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:39.632480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:39.632513 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:39.632750 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.633520 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.635062 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.635620 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.635670 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.635679 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:39.635955 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 04:18:39.636003 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:18:39.636550 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.631894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:45741 (local address 127.15.216.65:53473)
0810 04:18:39.632277 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:39.632288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:39.632384 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:39.632588 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:39.632589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:39.632591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:39.632668 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.632670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.633651 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.633652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.634937 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.634940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.635761 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.635762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.635831 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.635835 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:18:39.636405 (+ 570us) client_negotiation.cc:240] Negotiation successful
0810 04:18:39.636450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":122,"threads_started":1}
I20260810 04:18:39.636691 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.632172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:36093 (local address 127.15.216.67:32953)
0810 04:18:39.632218 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:39.632220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:39.632375 (+ 155us) server_negotiation.cc:407] Connection header received
0810 04:18:39.632896 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:39.632897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:39.632908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:39.632939 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:39.633737 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.634414 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.635715 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.636245 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.636288 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.636294 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:39.636586 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 04:18:39.636626 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:18:39.636884 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.631958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:32953 (local address 127.15.216.65:36093)
0810 04:18:39.632303 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:39.632314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:39.632406 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:39.633010 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:39.633011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:39.633012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:39.633635 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.633639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.634515 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.634515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.635610 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:39.635613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.636351 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.636351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:39.636506 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.636510 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:18:39.636774 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 04:18:39.636813 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":89,"threads_started":1}
I20260810 04:18:39.637346 30147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bc6a8a30c24d90b32344080cd02799" candidate_uuid: "c9abbbd3e47c4440b122f7e356a92eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6205d7092dc40069ed0824007d26731" is_pre_election: true
I20260810 04:18:39.637535 30147 raft_consensus.cc:2454] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9abbbd3e47c4440b122f7e356a92eba in term 0.
I20260810 04:18:39.637719 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bc6a8a30c24d90b32344080cd02799" candidate_uuid: "c9abbbd3e47c4440b122f7e356a92eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536ebb41ecde40e4ae298c1f11184120" is_pre_election: true
I20260810 04:18:39.637787 29932 leader_election.cc:302] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6205d7092dc40069ed0824007d26731, c9abbbd3e47c4440b122f7e356a92eba; no voters:
I20260810 04:18:39.637885 30288 raft_consensus.cc:2454] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9abbbd3e47c4440b122f7e356a92eba in term 0.
I20260810 04:18:39.637915 30379 raft_consensus.cc:2781] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:18:39.637954 30379 raft_consensus.cc:482] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:18:39.637975 30379 raft_consensus.cc:3037] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:39.638895 30379 raft_consensus.cc:504] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.639045 30379 leader_election.cc:288] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [CANDIDATE]: Term 1 election: Requested vote from peers c6205d7092dc40069ed0824007d26731 (127.15.216.66:45741), 536ebb41ecde40e4ae298c1f11184120 (127.15.216.67:32953)
I20260810 04:18:39.639271 30288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bc6a8a30c24d90b32344080cd02799" candidate_uuid: "c9abbbd3e47c4440b122f7e356a92eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536ebb41ecde40e4ae298c1f11184120"
I20260810 04:18:39.639305 30147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bc6a8a30c24d90b32344080cd02799" candidate_uuid: "c9abbbd3e47c4440b122f7e356a92eba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6205d7092dc40069ed0824007d26731"
I20260810 04:18:39.639357 30288 raft_consensus.cc:3037] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:39.639382 30147 raft_consensus.cc:3037] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:39.640233 30147 raft_consensus.cc:2454] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9abbbd3e47c4440b122f7e356a92eba in term 1.
I20260810 04:18:39.640309 30288 raft_consensus.cc:2454] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9abbbd3e47c4440b122f7e356a92eba in term 1.
I20260810 04:18:39.640486 29933 leader_election.cc:302] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536ebb41ecde40e4ae298c1f11184120, c9abbbd3e47c4440b122f7e356a92eba; no voters:
I20260810 04:18:39.640614 30379 raft_consensus.cc:2781] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:18:39.640802 30379 raft_consensus.cc:686] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [term 1 LEADER]: Becoming Leader. State: Replica: c9abbbd3e47c4440b122f7e356a92eba, State: Running, Role: LEADER
I20260810 04:18:39.640918 30379 consensus_queue.cc:237] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } }
I20260810 04:18:39.641880 29841 catalog_manager.cc:5495] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba reported cstate change: term changed from 0 to 1, leader changed from <none> to c9abbbd3e47c4440b122f7e356a92eba (127.15.216.65). New cstate: current_term: 1 leader_uuid: "c9abbbd3e47c4440b122f7e356a92eba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9abbbd3e47c4440b122f7e356a92eba" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 } health_report { overall_health: UNKNOWN } } }
I20260810 04:18:39.654915 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.651638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.15.216.65:35373)
0810 04:18:39.651693 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:39.651695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:39.651736 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:18:39.651778 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:39.651779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:39.651800 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:39.651875 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:18:39.652057 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.652784 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.653944 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:39.654223 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:39.654291 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:39.654494 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:18:39.654628 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:18:39.654708 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:39.654806 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 04:18:39.654830 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 04:18:39.656987 30147 raft_consensus.cc:1261] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 1 FOLLOWER]: Refusing update from remote peer c9abbbd3e47c4440b122f7e356a92eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:39.657399 30379 consensus_queue.cc:1035] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6205d7092dc40069ed0824007d26731" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:39.657974 30288 raft_consensus.cc:1261] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 1 FOLLOWER]: Refusing update from remote peer c9abbbd3e47c4440b122f7e356a92eba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:39.658458 30379 consensus_queue.cc:1035] T 09bc6a8a30c24d90b32344080cd02799 P c9abbbd3e47c4440b122f7e356a92eba [LEADER]: Connected to new peer: Peer: permanent_uuid: "536ebb41ecde40e4ae298c1f11184120" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:39.661116 30387 mvcc.cc:205] Tried to move back new op lower bound from 7316830288512151552 to 7316830288449863680. Current Snapshot: MvccSnapshot[applied={T|T < 7316830288512151552}]
I20260810 04:18:39.663522 30391 mvcc.cc:205] Tried to move back new op lower bound from 7316830288512151552 to 7316830288449863680. Current Snapshot: MvccSnapshot[applied={T|T < 7316830288512151552}]
I20260810 04:18:39.665283 29841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:18:39.665397 29841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:18:39.666143 29841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 419315322015439ba074a21a52261923: Sending DeleteTablet for 3 replicas of tablet 09bc6a8a30c24d90b32344080cd02799
I20260810 04:18:39.666575 30125 tablet_service.cc:1514] Processing DeleteTablet for tablet 09bc6a8a30c24d90b32344080cd02799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:39 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:41894
I20260810 04:18:39.667001 30268 tablet_service.cc:1514] Processing DeleteTablet for tablet 09bc6a8a30c24d90b32344080cd02799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:39 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:54062
I20260810 04:18:39.667577 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 29880
I20260810 04:18:39.669524 30388 mvcc.cc:205] Tried to move back new op lower bound from 7316830288512151552 to 7316830288449863680. Current Snapshot: MvccSnapshot[applied={T|T < 7316830288512151552}]
I20260810 04:18:39.672132 30399 tablet_replica.cc:330] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: stopping tablet replica
I20260810 04:18:39.672261 30399 raft_consensus.cc:2229] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:18:39.672374 30399 raft_consensus.cc:2258] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:18:39.672724 30398 tablet_replica.cc:330] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: stopping tablet replica
I20260810 04:18:39.672830 30398 raft_consensus.cc:2229] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:18:39.672925 30398 raft_consensus.cc:2258] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:18:39.673393 30399 ts_tablet_manager.cc:1905] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:18:39.678555 30399 ts_tablet_manager.cc:1918] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:18:39.678656 30399 log.cc:1196] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/09bc6a8a30c24d90b32344080cd02799
I20260810 04:18:39.678956 30399 ts_tablet_manager.cc:1939] T 09bc6a8a30c24d90b32344080cd02799 P 536ebb41ecde40e4ae298c1f11184120: Deleting consensus metadata
I20260810 04:18:39.679423 29825 catalog_manager.cc:4841] TS 536ebb41ecde40e4ae298c1f11184120 (127.15.216.67:32953): tablet 09bc6a8a30c24d90b32344080cd02799 (table test-table [id=08d21bee494749019798adbf60657b92]) successfully deleted
W20260810 04:18:39.679641 29827 connection.cc:535] client connection to 127.15.216.65:35373 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 04:18:39.679728 29827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:18:39.679983 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30054
I20260810 04:18:39.680203 30398 ts_tablet_manager.cc:1905] T 09bc6a8a30c24d90b32344080cd02799 P c6205d7092dc40069ed0824007d26731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:18:39.681571 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.681202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:35373 (local address 127.0.0.1:48444)
0810 04:18:39.681462 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:39.681524 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
W20260810 04:18:39.681720 29827 catalog_manager.cc:4584] TS c9abbbd3e47c4440b122f7e356a92eba (127.15.216.65:35373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09bc6a8a30c24d90b32344080cd02799: Network error: Client connection negotiation failed: client connection to 127.15.216.65:35373: connect: Connection refused (error 111)
I20260810 04:18:39.688268 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30200
I20260810 04:18:39.688294 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:39.688189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:45741 (local address 127.0.0.1:41900)
0810 04:18:39.688235 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:39.688257 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.66:45741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
Aug 10 04:18:39 dist-test-slave-g350 krb5kdc[28767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for HTTP/127.15.216.84@KRBTEST.COM
I20260810 04:18:39.697549 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 29806
I20260810 04:18:39.711040 16225 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 04:18:40.056003 16225 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 04:18:40.056102 16225 mini_ranger.cc:66] Stopping Ranger...
I20260810 04:18:40.410832 16225 mini_ranger.cc:68] Stopped Ranger
2026-08-10 04:18:40.411 UTC [28798] LOG: received smart shutdown request
2026-08-10 04:18:40.415 UTC [28798] LOG: background worker "logical replication launcher" (PID 28807) exited with exit code 1
2026-08-10 04:18:40.415 UTC [28802] LOG: shutting down
2026-08-10 04:18:40.609 UTC [28798] LOG: database system is shut down
2026-08-10T04:18:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50204 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:18:40 dist-test-slave-g350 krb5kdc[30410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:18:40 dist-test-slave-g350 krb5kdc[30410](info): set up 2 sockets
Aug 10 04:18:40 dist-test-slave-g350 krb5kdc[30410](info): commencing operation
krb5kdc: starting...
W20260810 04:18:41.852291 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:18:41 dist-test-slave-g350 krb5kdc[30410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:18:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:18:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:18:41.988686 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39917
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:39917,127.15.216.125:34707,127.15.216.124:37311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:42.073186 30423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:42.073428 30423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:42.073467 30423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:42.074599 30423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:42.074671 30423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:42.074707 30423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:42.074739 30423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:42.074769 30423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:42.076274 30423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:39917,127.15.216.125:34707,127.15.216.124:37311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:42.076555 30423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:42.076792 30423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:42.082017 30428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:42.082017 30432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:42.082124 30423 server_base.cc:1034] running on GCE node
W20260810 04:18:42.082019 30429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:42.082679 30423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:42.082954 30423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:42.084095 30423 hybrid_clock.cc:648] HybridClock initialized: now 1786335522084027 us; error 92 us; skew 500 ppm
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:42.089991 30423 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:18:42.090549 30423 webserver.cc:460] Webserver started at http://127.15.216.126:39245/ using document root <none> and password file <none>
I20260810 04:18:42.090806 30423 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:42.090855 30423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:42.090991 30423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:42.092088 30423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "bfbde55b0a7a46c696726f501cfa088e"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "a3b59c494905edb44c12c42febe1aa43"
server_key_iv: "048f93008365bef8a7704953581b8a2b"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.092464 30423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "bfbde55b0a7a46c696726f501cfa088e"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "a3b59c494905edb44c12c42febe1aa43"
server_key_iv: "048f93008365bef8a7704953581b8a2b"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.094156 30423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:18:42.094976 30439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:42.095170 30423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:42.095323 30423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "bfbde55b0a7a46c696726f501cfa088e"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "a3b59c494905edb44c12c42febe1aa43"
server_key_iv: "048f93008365bef8a7704953581b8a2b"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.095403 30423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:42.121042 30423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:42.123358 30423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:42.123548 30423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:42.127985 30423 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:39917
I20260810 04:18:42.128021 30497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:39917 every 8 connection(s)
I20260810 04:18:42.128410 30423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:18:42.129398 30498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:42.131142 30498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:18:42.134173 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30423
I20260810 04:18:42.134310 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:18:42.134542 16225 external_mini_cluster.cc:1428] Setting key 899fb663632fc79e6638ee05c1cb8069
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:42.141554 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.131823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39917 (local address 127.0.0.1:42212)
0810 04:18:42.132087 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.132133 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.132653 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.132975 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.132978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.133262 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.133463 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.133468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.134670 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.134671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.135383 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.135385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.135900 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.135901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.135977 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.136557 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.136565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.138208 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.140059 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.140066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.140068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.140266 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.140513 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.140514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.140514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.140608 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.140961 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.140966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.141028 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.141143 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.141389 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":106,"thread_start_us":113,"threads_started":1}
I20260810 04:18:42.141566 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.131912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.15.216.126:39917)
0810 04:18:42.132098 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.132108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.132243 (+ 135us) server_negotiation.cc:407] Connection header received
0810 04:18:42.132848 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.132850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.132867 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.132970 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.133573 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.134565 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.135480 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.135811 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.135942 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.138316 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.138323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.138324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.138337 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.139953 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.140352 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.140355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.140356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.140440 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.140675 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.140676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.140677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.140887 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.140981 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.141170 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.141320 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260810 04:18:42.142699 30498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.216.125@KRBTEST.COM; defaulting to no policy
I20260810 04:18:42.143074 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.142948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37674)
0810 04:18:42.142998 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.143043 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:18:42.143229 30443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
I20260810 04:18:42.143648 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.143558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37676)
0810 04:18:42.143606 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.143632 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:18:42.143762 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:34707: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab.
Entry for principal kudu/127.15.216.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab.
I20260810 04:18:42.164763 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:34707
--webserver_interface=127.15.216.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab
--principal=kudu/127.15.216.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:39917,127.15.216.125:34707,127.15.216.124:37311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:42.193058 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } attempt: 1
I20260810 04:18:42.193764 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.193501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37680)
0810 04:18:42.193678 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.193727 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 04:18:42.194244 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.194090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37688)
0810 04:18:42.194144 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.194174 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 04:18:42.194454 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:34707: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
W20260810 04:18:42.249702 30506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:42.249943 30506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:42.249965 30506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:42.251101 30506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:42.251147 30506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:42.251160 30506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:42.251173 30506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:42.251184 30506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:42.252667 30506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:39917,127.15.216.125:34707,127.15.216.124:37311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab
--principal=kudu/127.15.216.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.216.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:42.252924 30506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:42.253160 30506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:42.258558 30511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:42.258572 30514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:42.258634 30512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:42.258831 30506 server_base.cc:1034] running on GCE node
I20260810 04:18:42.259073 30506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:42.259305 30506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:42.260457 30506 hybrid_clock.cc:648] HybridClock initialized: now 1786335522260442 us; error 28 us; skew 500 ppm
I20260810 04:18:42.262624 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } attempt: 2
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:42.263036 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.262913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37702)
0810 04:18:42.262971 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.263007 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:18:42.263289 30506 init.cc:377] Logged in from keytab as kudu/127.15.216.125@KRBTEST.COM (short username kudu)
I20260810 04:18:42.263353 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.263263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37712)
0810 04:18:42.263308 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.263332 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 04:18:42.263511 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:34707: Network error: Client connection negotiation failed: client connection to 127.15.216.125:34707: connect: Connection refused (error 111)
I20260810 04:18:42.263828 30506 webserver.cc:460] Webserver started at http://127.15.216.125:44601/ using document root <none> and password file <none>
I20260810 04:18:42.264074 30506 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:42.264122 30506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:42.264317 30506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:42.265470 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data/instance:
uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "c02ecc660b3993d54918f2905b5cdc0c"
server_key_iv: "1dbf901a5b715b9a9346702fee082725"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.265883 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/wal/instance:
uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "c02ecc660b3993d54918f2905b5cdc0c"
server_key_iv: "1dbf901a5b715b9a9346702fee082725"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.267349 30506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:42.268098 30523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:42.268379 30506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:42.268476 30506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/wal
uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "c02ecc660b3993d54918f2905b5cdc0c"
server_key_iv: "1dbf901a5b715b9a9346702fee082725"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.268530 30506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:42.287997 30506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:42.290143 30506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:42.290271 30506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:42.294407 30506 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.125:34707
I20260810 04:18:42.294466 30579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.125:34707 every 8 connection(s)
I20260810 04:18:42.294826 30506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
I20260810 04:18:42.295738 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:42.297309 30580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:18:42.300561 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30506
I20260810 04:18:42.300725 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-1/wal/instance
I20260810 04:18:42.300940 16225 external_mini_cluster.cc:1428] Setting key ea04e64c2113b9ff6332d8ba7176f626
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:42.307847 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.298102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39917 (local address 127.0.0.1:42220)
0810 04:18:42.298282 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.298319 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.298819 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.299165 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.299169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.299444 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.299655 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.299659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.300386 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.300387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.301378 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.301381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.301870 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.301871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.302010 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.302581 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.302590 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.304391 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.306425 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.306433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.306434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.306657 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.306906 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.306906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.306906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.307005 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.307323 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.307328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.307471 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.307556 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.307681 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260810 04:18:42.307884 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.298112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.15.216.126:39917)
0810 04:18:42.298179 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.298182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.298411 (+ 229us) server_negotiation.cc:407] Connection header received
0810 04:18:42.298936 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.298938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.298954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.299051 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.299756 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.300264 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.301467 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.301758 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.301846 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.304499 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.304500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.304501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.304510 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.306338 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.306746 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.306748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.306748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.306822 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.307072 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.307073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.307074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.307247 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.307339 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.307611 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.307730 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:18:42.308450 30580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.216.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.124@KRBTEST.COM" created.
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:18:42.318022 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.308747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37724)
0810 04:18:42.308809 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.308818 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.308864 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.309138 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.309138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.309286 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.309335 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.309336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.310269 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.310270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.310830 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.310831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.311338 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.311339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.311408 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.311831 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.311836 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.314412 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.316761 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.316762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.316762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.316942 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.317205 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.317206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.317207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.317268 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.317638 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.317639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.317685 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.317776 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.317893 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 04:18:42.318099 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.308819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.15.216.125:34707)
0810 04:18:42.309004 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.309007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.309024 (+ 17us) server_negotiation.cc:407] Connection header received
0810 04:18:42.309041 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.309042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.309052 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.309137 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.309424 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.310178 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.310928 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.311263 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.311341 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.314521 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.314532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.314533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.314543 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.316544 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.317061 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.317064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.317065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.317128 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.317355 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.317356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.317356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.317520 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.317669 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.317818 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.317954 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260810 04:18:42.319008 30580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:18:42.319404 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.319269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51084)
0810 04:18:42.319329 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.319378 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 04:18:42.319581 30527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
I20260810 04:18:42.320008 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.319936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51094)
0810 04:18:42.319965 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.319990 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260810 04:18:42.320135 30580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:37311: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab.
Entry for principal kudu/127.15.216.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab.
I20260810 04:18:42.330834 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:37311
--webserver_interface=127.15.216.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab
--principal=kudu/127.15.216.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:39917,127.15.216.125:34707,127.15.216.124:37311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:42.354734 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } attempt: 3
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:18:42.363193 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.355473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.15.216.125:34707)
0810 04:18:42.355550 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.355553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.355567 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:18:42.355599 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.355600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.355619 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.355696 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.356092 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.356619 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.357447 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.357760 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.357850 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.360255 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.360257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.360257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.360265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.361915 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.362317 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.362318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.362319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.362384 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.362608 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.362609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.362609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.362744 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.362831 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.362996 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.363084 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:18:42.363193 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.355220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37730)
0810 04:18:42.355408 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.355424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.355520 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.355717 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.355719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.355921 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.355994 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.355996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.356756 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.356758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.357342 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.357344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.357863 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.357864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.357956 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.358487 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.358499 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.360099 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.362025 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.362026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.362026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.362221 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.362467 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.362468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.362468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.362540 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.362822 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.362824 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.362877 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.362968 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.363082 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260810 04:18:42.363854 30498 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:18:42.364259 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.364137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51102)
0810 04:18:42.364195 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.364226 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 04:18:42.364596 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.364502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51108)
0810 04:18:42.364548 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.364573 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 04:18:42.364714 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:37311: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
I20260810 04:18:42.369369 30580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } attempt: 1
I20260810 04:18:42.369807 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.369672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51112)
0810 04:18:42.369738 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.369769 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 04:18:42.370163 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.370064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51124)
0810 04:18:42.370111 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.370138 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 04:18:42.370278 30580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:37311: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
I20260810 04:18:42.395893 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } attempt: 1
I20260810 04:18:42.396447 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.396295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51126)
0810 04:18:42.396375 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.396415 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 04:18:42.396821 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.396705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51130)
0810 04:18:42.396768 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.396792 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 04:18:42.396914 30498 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:37311: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
W20260810 04:18:42.416085 30587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:42.416281 30587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:42.416303 30587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:42.417418 30587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:42.417461 30587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:42.417475 30587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:42.417487 30587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:42.417500 30587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:42.419046 30587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:39917,127.15.216.125:34707,127.15.216.124:37311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab
--principal=kudu/127.15.216.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:37311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.216.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:42.419301 30587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:42.419545 30587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:42.424628 30593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:42.424598 30595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:42.424733 30587 server_base.cc:1034] running on GCE node
W20260810 04:18:42.424623 30592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:42.425141 30587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:42.425419 30587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:42.426559 30587 hybrid_clock.cc:648] HybridClock initialized: now 1786335522426551 us; error 48 us; skew 500 ppm
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:42.429107 30587 init.cc:377] Logged in from keytab as kudu/127.15.216.124@KRBTEST.COM (short username kudu)
I20260810 04:18:42.429543 30587 webserver.cc:460] Webserver started at http://127.15.216.124:37887/ using document root <none> and password file <none>
I20260810 04:18:42.429808 30587 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:42.429880 30587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:42.430028 30587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:42.431119 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data/instance:
uuid: "88db2bee420b4fe88b49d7e40ac41e1b"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "9cb8f640dd7708fcf2d7f18e9c01b68a"
server_key_iv: "d2e61f2912004d793c55922203ee8e72"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.431484 30587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/wal/instance:
uuid: "88db2bee420b4fe88b49d7e40ac41e1b"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "9cb8f640dd7708fcf2d7f18e9c01b68a"
server_key_iv: "d2e61f2912004d793c55922203ee8e72"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.432824 30587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:18:42.433534 30602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:42.433713 30587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:18:42.433831 30587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/wal
uuid: "88db2bee420b4fe88b49d7e40ac41e1b"
format_stamp: "Formatted at 2026-08-10 04:18:42 on dist-test-slave-g350"
server_key: "9cb8f640dd7708fcf2d7f18e9c01b68a"
server_key_iv: "d2e61f2912004d793c55922203ee8e72"
server_key_version: "encryptionkey@0"
I20260810 04:18:42.433907 30587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:42.438421 30580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } attempt: 2
I20260810 04:18:42.438794 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.438668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51136)
0810 04:18:42.438727 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.438764 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:18:42.439244 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.439148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51150)
0810 04:18:42.439191 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.439221 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:18:42.439406 30580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:37311: Network error: Client connection negotiation failed: client connection to 127.15.216.124:37311: connect: Connection refused (error 111)
I20260810 04:18:42.449079 30587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:42.451478 30587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:42.451645 30587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:42.455895 30587 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.124:37311
I20260810 04:18:42.456254 30663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.124:37311 every 8 connection(s)
I20260810 04:18:42.456279 30587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
I20260810 04:18:42.457021 30664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:42.458671 30664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:42.466415 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30587
I20260810 04:18:42.466537 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786335362448781-16225-0/minicluster-data/master-2/wal/instance
I20260810 04:18:42.466750 16225 external_mini_cluster.cc:1428] Setting key b692dc6af75d22d6d8fddba4b62b9ca0
I20260810 04:18:42.468045 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.459465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.15.216.126:39917)
0810 04:18:42.459556 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.459559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.459715 (+ 156us) server_negotiation.cc:407] Connection header received
0810 04:18:42.460294 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.460296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.460309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.460357 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.460991 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.461466 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.462512 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.462731 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.462809 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.464857 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.464859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.464859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.464867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.466445 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.466965 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.466966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.466967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.467054 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.467352 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.467353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.467353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.467511 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.467579 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.467914 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.467981 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 04:18:42.468330 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.459354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:39917 (local address 127.0.0.1:42234)
0810 04:18:42.459594 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.459629 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.460122 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.460433 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.460436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.460685 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.460873 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.460877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.461563 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.461564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.462417 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.462420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.462829 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.462830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.462959 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.463396 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.463404 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.464759 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.466553 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.466579 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.466582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.466885 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.467123 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.467124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.467125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.467263 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.467587 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.467595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.467758 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.467883 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.468056 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":121,"threads_started":1}
I20260810 04:18:42.468936 30664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:18:42.472033 30498 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } attempt: 2
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:18:42.476450 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.469278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.15.216.125:34707)
0810 04:18:42.469327 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.469329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.469337 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:18:42.469369 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.469370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.469379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.469434 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.469761 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.470343 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.471038 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.471400 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.471481 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.473483 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.473484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.473485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.473492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.475219 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.475594 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.475595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.475595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.475663 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.475855 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.475856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.475856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.475988 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.476070 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.476257 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.476341 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:18:42.476505 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.467140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.15.216.124:37311)
0810 04:18:42.467323 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.467327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.467356 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:18:42.467390 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.467392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.467406 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.467493 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.468232 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.469125 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.469966 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.470260 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.470341 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.473020 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.473028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.473029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.473035 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.474999 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.475459 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.475461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.475462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.475517 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.475928 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.475930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.475930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.476098 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.476273 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.476323 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.476401 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260810 04:18:42.477078 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.469213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:34707 (local address 127.0.0.1:37736)
0810 04:18:42.469254 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.469262 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.469295 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.469442 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.469443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.469578 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.469676 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.469677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.470485 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.470486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.470958 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.470960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.471490 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.471491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.471557 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.471946 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.471950 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.473421 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.475315 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.475316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.475317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.475525 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.475730 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.475730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.475731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.475782 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.476064 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.476065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.476116 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.476868 (+ 752us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.476958 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 04:18:42.477555 30664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
I20260810 04:18:42.482405 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.473276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51174)
0810 04:18:42.473327 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.473336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.473372 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.474389 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.474390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.474560 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.474596 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.474597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.475414 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.475415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.476158 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.476160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.476846 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.476847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.476905 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.477398 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.477403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.478879 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.481324 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.481325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.481326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.481512 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.481751 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.481752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.481752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.481816 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.482079 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.482080 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.482125 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.482213 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.482309 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:18:42.482435 30670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.473671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.15.216.124:37311)
0810 04:18:42.474240 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.474243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.474251 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:18:42.474272 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.474272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.474283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.474370 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.474676 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.475324 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.476245 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.476539 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.477054 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.478961 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.478963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.478963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.478969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.481238 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.481588 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.481609 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.481610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.481669 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.481898 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.481899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.481899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.482009 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.482086 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.482240 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.482312 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":110,"threads_started":1}
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:42.485141 30498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: Bootstrap starting.
I20260810 04:18:42.485962 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.477348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.15.216.126:39917)
0810 04:18:42.477388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.477389 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.477457 (+ 68us) server_negotiation.cc:407] Connection header received
0810 04:18:42.477542 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.477543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.477552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.477579 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.478282 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.478955 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.479596 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.479889 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.479954 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.483141 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.483142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.483142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.483150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.484773 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.485137 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.485138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.485139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.485248 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.485453 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.485454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.485454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.485572 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.485760 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.485844 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.485904 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:18:42.486099 30498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:42.486459 30498 log.cc:824] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:42.486900 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.477801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51188)
0810 04:18:42.477917 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.477927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.477981 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.478160 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.478160 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.478312 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.478346 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.478347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.479089 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.479089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.479842 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.479845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.480597 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.480599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.480660 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.480962 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.480967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.483190 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.485184 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.485185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.485186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.485381 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.486192 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.486193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.486193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.486253 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.486558 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.486559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.486604 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.486686 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.486797 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:18:42.487038 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.477886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51188 (local address 127.15.216.124:37311)
0810 04:18:42.477924 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.477926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.477967 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:18:42.478051 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.478051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.478061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.478102 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.478434 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.479006 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.480191 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.480513 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.480582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.483263 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.483264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.483264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.483271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.485094 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.486054 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.486055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.486056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.486115 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.486328 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.486329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.486329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.486473 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.486562 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.486747 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.486959 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":19}
I20260810 04:18:42.487664 30498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: No bootstrap required, opened a new log
I20260810 04:18:42.488704 30664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: Bootstrap starting.
I20260810 04:18:42.489198 30498 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.489334 30498 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:42.489398 30498 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbde55b0a7a46c696726f501cfa088e, State: Initialized, Role: FOLLOWER
I20260810 04:18:42.489425 30664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:42.489548 30498 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.489773 30664 log.cc:824] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:42.489859 30672 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.489926 30498 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:42.489929 30672 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:18:42.490669 30664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: No bootstrap required, opened a new log
W20260810 04:18:42.491779 30684 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:18:42.491878 30684 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:18:42.492703 30664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.492851 30664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:42.492897 30664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88db2bee420b4fe88b49d7e40ac41e1b, State: Initialized, Role: FOLLOWER
I20260810 04:18:42.493024 30664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.493283 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.493340 30664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:42.493425 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:18:42.493207 30423 master_runner.cc:186] Error getting master registration for 127.15.216.125:34707: OK, instance_id { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" instance_seqno: 1786335522293189 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:18:42.496502 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.487353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.15.216.125:34707)
0810 04:18:42.487400 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.487402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.487413 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:18:42.487441 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.487442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.487451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.487496 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.488098 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.488806 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.489828 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.490115 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.490168 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.492888 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.492889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.492889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.492897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.494846 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.495304 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.495306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.495306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.495366 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.495568 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.495569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.495569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.495701 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.495867 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.496334 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.496415 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:18:42.496562 30423 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 04:18:42.495559 30698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:18:42.497330 30698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:18:42.499030 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:18:42.500623 30587 master_runner.cc:186] Error getting master registration for 127.15.216.125:34707: OK, instance_id { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" instance_seqno: 1786335522293189 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:18:42.503090 30587 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:18:42.510660 30502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.501670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.15.216.126:39917)
0810 04:18:42.501712 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.501714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.501789 (+ 75us) server_negotiation.cc:407] Connection header received
0810 04:18:42.501908 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.501909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.501922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.501953 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.503753 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.504639 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.505729 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.506026 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.506068 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.507637 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.507639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.507639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.507646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.509332 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.509773 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.509774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.509775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.509831 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.510025 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.510025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.510026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.510150 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.510261 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.510477 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.510557 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:18:42.511224 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.500789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.15.216.124:37311)
0810 04:18:42.500840 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.500842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.502782 (+ 1940us) server_negotiation.cc:407] Connection header received
0810 04:18:42.502892 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.502893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.502903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.502933 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.503739 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.504473 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.505635 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.505926 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.505971 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.507764 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.507766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.507766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.507773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.509942 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.510330 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.510331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.510332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.510388 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.510586 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.510586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.510587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.510707 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.510784 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.511027 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.511110 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:18:42.518965 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.503392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.15.216.125:34707)
0810 04:18:42.503429 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.503430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.509780 (+ 6350us) server_negotiation.cc:407] Connection header received
0810 04:18:42.511672 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.511673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.511685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.511734 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.512415 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.513138 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.514176 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.514492 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.514532 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.516024 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.516026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.516026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.516034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.517705 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.518108 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.518109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.518110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.518165 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.518372 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.518373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.518373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.518497 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.518579 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.518790 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.518868 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:18:42.530579 30580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } attempt: 3
Aug 10 04:18:42 dist-test-slave-g350 krb5kdc[30410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
I20260810 04:18:42.538916 30582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.530870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:37311 (local address 127.0.0.1:51214)
0810 04:18:42.530923 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:42.530935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:42.530971 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:42.531111 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:42.531112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:42.531322 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:42.531377 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.531379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.532370 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.532371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.533148 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:42.533150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.533676 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.533677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:42.533740 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.534258 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:42.534264 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:42.535827 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:42.537798 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.537799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.537800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.538022 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.538248 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:42.538249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:42.538249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:42.538317 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:42.538588 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:42.538589 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:42.538637 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 04:18:42.538708 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:18:42.538808 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:18:42.538938 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:42.530939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.15.216.124:37311)
0810 04:18:42.530981 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:42.530983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:42.530993 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:18:42.531018 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:42.531019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:42.531028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:42.531091 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:42.531458 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.532197 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.533285 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:42.533606 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:42.533673 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:42.535904 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:42.535905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:42.535906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:42.535923 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:42.537713 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.538104 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.538105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.538105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.538168 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:42.538384 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:42.538384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:42.538385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:42.538518 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:42.538588 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:42.538759 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 04:18:42.538842 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:18:42.541249 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec: Bootstrap starting.
I20260810 04:18:42.542071 30580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:42.542418 30580 log.cc:824] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:42.543354 30580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec: No bootstrap required, opened a new log
I20260810 04:18:42.545464 30580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.545637 30580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:42.545711 30580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0020f0b2fef941b6a35e0bd7e1ce50ec, State: Initialized, Role: FOLLOWER
I20260810 04:18:42.545866 30580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.546204 30580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:42.546173 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.546372 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:18:42.548178 30721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:18:42.548292 30721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:18:42.550488 30506 master_runner.cc:263] No leader master found from master 0020f0b2fef941b6a35e0bd7e1ce50ec
I20260810 04:18:42.550534 30506 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:18:42.610883 30709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:42.611022 30709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.611299 30709 leader_election.cc:288] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfbde55b0a7a46c696726f501cfa088e (127.15.216.126:39917), 88db2bee420b4fe88b49d7e40ac41e1b (127.15.216.124:37311)
I20260810 04:18:42.611948 30470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbde55b0a7a46c696726f501cfa088e" is_pre_election: true
I20260810 04:18:42.612098 30470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0020f0b2fef941b6a35e0bd7e1ce50ec in term 0.
I20260810 04:18:42.612555 30527 leader_election.cc:302] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0020f0b2fef941b6a35e0bd7e1ce50ec, bfbde55b0a7a46c696726f501cfa088e; no voters:
I20260810 04:18:42.612763 30709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:18:42.612821 30709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:18:42.612844 30709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:42.613972 30634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" is_pre_election: true
I20260810 04:18:42.614130 30634 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0020f0b2fef941b6a35e0bd7e1ce50ec in term 0.
I20260810 04:18:42.614734 30709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.614853 30709 leader_election.cc:288] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [CANDIDATE]: Term 1 election: Requested vote from peers bfbde55b0a7a46c696726f501cfa088e (127.15.216.126:39917), 88db2bee420b4fe88b49d7e40ac41e1b (127.15.216.124:37311)
I20260810 04:18:42.615136 30470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbde55b0a7a46c696726f501cfa088e"
I20260810 04:18:42.615223 30470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:42.616246 30470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0020f0b2fef941b6a35e0bd7e1ce50ec in term 1.
I20260810 04:18:42.616466 30527 leader_election.cc:302] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0020f0b2fef941b6a35e0bd7e1ce50ec, bfbde55b0a7a46c696726f501cfa088e; no voters:
I20260810 04:18:42.616694 30634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88db2bee420b4fe88b49d7e40ac41e1b"
I20260810 04:18:42.616806 30634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:42.617914 30634 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0020f0b2fef941b6a35e0bd7e1ce50ec in term 1.
I20260810 04:18:42.618163 30709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:18:42.618336 30709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [term 1 LEADER]: Becoming Leader. State: Replica: 0020f0b2fef941b6a35e0bd7e1ce50ec, State: Running, Role: LEADER
I20260810 04:18:42.618445 30709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } }
I20260810 04:18:42.618829 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0020f0b2fef941b6a35e0bd7e1ce50ec. Latest consensus state: current_term: 1 leader_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.618894 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:42.619208 30725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:18:42.619459 30725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:18:42.621803 30634 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [term 1 FOLLOWER]: Refusing update from remote peer 0020f0b2fef941b6a35e0bd7e1ce50ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:42.622066 30470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [term 1 FOLLOWER]: Refusing update from remote peer 0020f0b2fef941b6a35e0bd7e1ce50ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:42.622494 30709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:42.622879 30709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:42.624130 30672 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0020f0b2fef941b6a35e0bd7e1ce50ec. Latest consensus state: current_term: 1 leader_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.624215 30672 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:18:42.624352 30445 tablet.cc:2369] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:18:42.626150 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0020f0b2fef941b6a35e0bd7e1ce50ec. Latest consensus state: current_term: 1 leader_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.626235 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:18:42.628132 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.628208 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:42.628708 30672 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.628787 30672 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:18:42.629040 30726 mvcc.cc:205] Tried to move back new op lower bound from 7316830300655951872 to 7316830300645670912. Current Snapshot: MvccSnapshot[applied={T|T < 7316830300655951872}]
I20260810 04:18:42.629256 30727 mvcc.cc:205] Tried to move back new op lower bound from 7316830300655951872 to 7316830300645670912. Current Snapshot: MvccSnapshot[applied={T|T < 7316830300655951872}]
I20260810 04:18:42.629685 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.629760 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:42.630151 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbde55b0a7a46c696726f501cfa088e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 39917 } } peers { permanent_uuid: "0020f0b2fef941b6a35e0bd7e1ce50ec" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 34707 } } peers { permanent_uuid: "88db2bee420b4fe88b49d7e40ac41e1b" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 37311 } } }
I20260810 04:18:42.630228 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:18:42.630820 30729 mvcc.cc:205] Tried to move back new op lower bound from 7316830300655951872 to 7316830300645670912. Current Snapshot: MvccSnapshot[applied={T|T < 7316830300655951872}]
I20260810 04:18:42.631865 30725 catalog_manager.cc:1347] Generated new cluster ID: 49505792da964a519a8723cc58ba78be
I20260810 04:18:42.631912 30725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:18:42.639660 30725 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:18:42.641275 30725 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:18:42.650787 30725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0020f0b2fef941b6a35e0bd7e1ce50ec: Generated new TSK 0
I20260810 04:18:42.651053 30725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:18:43.492388 30684 catalog_manager.cc:1259] Loaded cluster ID: 49505792da964a519a8723cc58ba78be
I20260810 04:18:43.492475 30684 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: loading cluster ID for follower catalog manager: success
I20260810 04:18:43.493417 30684 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: acquiring CA information for follower catalog manager: success
I20260810 04:18:43.493777 30684 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfbde55b0a7a46c696726f501cfa088e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:18:43.497817 30698 catalog_manager.cc:1259] Loaded cluster ID: 49505792da964a519a8723cc58ba78be
I20260810 04:18:43.497907 30698 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: loading cluster ID for follower catalog manager: success
I20260810 04:18:43.498696 30698 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: acquiring CA information for follower catalog manager: success
I20260810 04:18:43.499037 30698 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88db2bee420b4fe88b49d7e40ac41e1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:18:43.703825 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30423
I20260810 04:18:43.712355 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30506
I20260810 04:18:43.720304 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30587
2026-08-10T04:18:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2988 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:18:43 dist-test-slave-g350 krb5kdc[30743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:18:43 dist-test-slave-g350 krb5kdc[30743](info): set up 2 sockets
Aug 10 04:18:43 dist-test-slave-g350 krb5kdc[30743](info): commencing operation
krb5kdc: starting...
W20260810 04:18:44.807710 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:18:44 dist-test-slave-g350 krb5kdc[30743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:18:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:18:44.941478 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40409
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:40409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:45.026296 30756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:45.026571 30756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:45.026611 30756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:45.027726 30756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:45.027765 30756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:45.027779 30756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:45.027792 30756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:45.027804 30756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:45.029240 30756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:40409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:45.029461 30756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:45.029703 30756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:45.034880 30765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:45.034885 30762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:45.035099 30756 server_base.cc:1034] running on GCE node
W20260810 04:18:45.034896 30761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:45.035490 30756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:45.035841 30756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:45.037029 30756 hybrid_clock.cc:648] HybridClock initialized: now 1786335525037017 us; error 32 us; skew 500 ppm
Aug 10 04:18:45 dist-test-slave-g350 krb5kdc[30743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335525, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:45.039747 30756 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:18:45.040243 30756 webserver.cc:460] Webserver started at http://127.15.216.126:39143/ using document root <none> and password file <none>
I20260810 04:18:45.040436 30756 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:45.040477 30756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:45.040566 30756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:45.041635 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "027dfbf20aa44c89b1347ed21f3775cc"
format_stamp: "Formatted at 2026-08-10 04:18:45 on dist-test-slave-g350"
server_key: "ab803955ffb429b780a9fee45f2c0109"
server_key_iv: "ce81f4963ada9be053ccf51a6bc7105d"
server_key_version: "encryptionkey@0"
I20260810 04:18:45.041989 30756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "027dfbf20aa44c89b1347ed21f3775cc"
format_stamp: "Formatted at 2026-08-10 04:18:45 on dist-test-slave-g350"
server_key: "ab803955ffb429b780a9fee45f2c0109"
server_key_iv: "ce81f4963ada9be053ccf51a6bc7105d"
server_key_version: "encryptionkey@0"
I20260810 04:18:45.043394 30756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:45.044206 30774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:45.044430 30756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:45.044530 30756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "027dfbf20aa44c89b1347ed21f3775cc"
format_stamp: "Formatted at 2026-08-10 04:18:45 on dist-test-slave-g350"
server_key: "ab803955ffb429b780a9fee45f2c0109"
server_key_iv: "ce81f4963ada9be053ccf51a6bc7105d"
server_key_version: "encryptionkey@0"
I20260810 04:18:45.044608 30756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:45.066278 30756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:45.068814 30756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:45.068984 30756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:45.073376 30756 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:40409
I20260810 04:18:45.073437 30828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:40409 every 8 connection(s)
I20260810 04:18:45.073910 30756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:18:45.074779 30829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:45.077262 30829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc: Bootstrap starting.
I20260810 04:18:45.077463 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30756
I20260810 04:18:45.077587 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:18:45.077838 16225 external_mini_cluster.cc:1428] Setting key 81aa137fd59e039daa83d4ce75062b23
I20260810 04:18:45.078073 30829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:45.078446 30829 log.cc:824] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:45.079222 30829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc: No bootstrap required, opened a new log
I20260810 04:18:45.081130 30829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027dfbf20aa44c89b1347ed21f3775cc" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40409 } }
I20260810 04:18:45.081290 30829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:45.081339 30829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027dfbf20aa44c89b1347ed21f3775cc, State: Initialized, Role: FOLLOWER
I20260810 04:18:45.081502 30829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027dfbf20aa44c89b1347ed21f3775cc" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40409 } }
I20260810 04:18:45.081588 30829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:18:45.081652 30829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:18:45.081702 30829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:45.082275 30829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027dfbf20aa44c89b1347ed21f3775cc" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40409 } }
I20260810 04:18:45.082408 30829 leader_election.cc:302] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 027dfbf20aa44c89b1347ed21f3775cc; no voters:
I20260810 04:18:45.082600 30829 leader_election.cc:288] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:18:45.082644 30834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:18:45.082872 30834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [term 1 LEADER]: Becoming Leader. State: Replica: 027dfbf20aa44c89b1347ed21f3775cc, State: Running, Role: LEADER
I20260810 04:18:45.082942 30829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:45.083030 30834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027dfbf20aa44c89b1347ed21f3775cc" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40409 } }
I20260810 04:18:45.083405 30835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "027dfbf20aa44c89b1347ed21f3775cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027dfbf20aa44c89b1347ed21f3775cc" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40409 } } }
I20260810 04:18:45.083488 30835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:45.083699 30836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 027dfbf20aa44c89b1347ed21f3775cc. Latest consensus state: current_term: 1 leader_uuid: "027dfbf20aa44c89b1347ed21f3775cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027dfbf20aa44c89b1347ed21f3775cc" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40409 } } }
I20260810 04:18:45.083810 30836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:45.084214 30842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:18:45.084703 30842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 04:18:45 dist-test-slave-g350 krb5kdc[30743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:45.086483 30842 catalog_manager.cc:1347] Generated new cluster ID: 3ad8e0aad9a640968e8afaba1199fd65
I20260810 04:18:45.086539 30842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:18:45.089965 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:45.078387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.15.216.126:40409)
0810 04:18:45.078610 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:45.078620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:45.078655 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:18:45.079141 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:45.079148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:45.079196 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:45.079372 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:45.080056 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:45.080967 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:45.081999 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:45.082474 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:45.082629 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:45.085572 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:45.085576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:45.085577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:45.085618 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:45.087767 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:45.088198 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:45.088201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:45.088201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:45.088309 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:45.088546 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:45.088546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:45.088547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:45.088861 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:45.089017 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:45.089560 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 04:18:45.089737 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":112,"threads_started":1}
I20260810 04:18:45.090994 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:18:45.094949 30842 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:18:45.095476 30842 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:18:45.098250 30842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 027dfbf20aa44c89b1347ed21f3775cc: Generated new TSK 0
I20260810 04:18:45.098484 30842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:18:45.145170 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30756
2026-08-10T04:18:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1423 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:18:45 dist-test-slave-g350 krb5kdc[30858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:18:45 dist-test-slave-g350 krb5kdc[30858](info): set up 2 sockets
Aug 10 04:18:45 dist-test-slave-g350 krb5kdc[30858](info): commencing operation
krb5kdc: starting...
W20260810 04:18:46.278486 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:18:46 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:18:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:18:46.412446 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:32953
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:46.496748 30872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:46.497012 30872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:46.497052 30872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:46.498348 30872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:46.498421 30872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:46.498447 30872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:46.498481 30872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:46.498507 30872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:46.500131 30872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:32953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:46.500425 30872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:46.500663 30872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:46.505921 30880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:46.505923 30878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:46.505921 30877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:46.506181 30872 server_base.cc:1034] running on GCE node
I20260810 04:18:46.506443 30872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:46.506716 30872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:46.507874 30872 hybrid_clock.cc:648] HybridClock initialized: now 1786335526507861 us; error 32 us; skew 500 ppm
Aug 10 04:18:46 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:46.510602 30872 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:18:46.511049 30872 webserver.cc:460] Webserver started at http://127.15.216.126:40213/ using document root <none> and password file <none>
I20260810 04:18:46.511312 30872 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:46.511386 30872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:46.511538 30872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:46.512614 30872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "61972f53d4b34f2d9d834eb231892574"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "bb06b5e29f340246f54f9de303d0e1ed"
server_key_iv: "62d98d8430c7896610bd27f7df42bc06"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.512989 30872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "61972f53d4b34f2d9d834eb231892574"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "bb06b5e29f340246f54f9de303d0e1ed"
server_key_iv: "62d98d8430c7896610bd27f7df42bc06"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.514487 30872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:18:46.515257 30887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:46.515455 30872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:46.515568 30872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "61972f53d4b34f2d9d834eb231892574"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "bb06b5e29f340246f54f9de303d0e1ed"
server_key_iv: "62d98d8430c7896610bd27f7df42bc06"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.515643 30872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:46.532320 30872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:46.534917 30872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:46.535094 30872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:46.539772 30872 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:32953
I20260810 04:18:46.539832 30946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:32953 every 8 connection(s)
I20260810 04:18:46.540208 30872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:18:46.541234 30947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:46.543689 30947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Bootstrap starting.
I20260810 04:18:46.544320 30947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:46.544616 30947 log.cc:824] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:46.545420 30947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: No bootstrap required, opened a new log
I20260810 04:18:46.547163 30947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:46.547312 30947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:46.547336 30947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61972f53d4b34f2d9d834eb231892574, State: Initialized, Role: FOLLOWER
I20260810 04:18:46.547528 30947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:46.547610 30947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:18:46.547659 30947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:18:46.547709 30947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:46.548609 30947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:46.548705 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30872
I20260810 04:18:46.548735 30947 leader_election.cc:302] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61972f53d4b34f2d9d834eb231892574; no voters:
I20260810 04:18:46.548820 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:18:46.548918 30947 leader_election.cc:288] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:18:46.548959 30950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:18:46.549047 16225 external_mini_cluster.cc:1428] Setting key 912c9fc8b51e286cdf65b7c929facbc7
I20260810 04:18:46.549154 30950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 LEADER]: Becoming Leader. State: Replica: 61972f53d4b34f2d9d834eb231892574, State: Running, Role: LEADER
I20260810 04:18:46.549273 30947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:46.549300 30950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:46.549896 30952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61972f53d4b34f2d9d834eb231892574. Latest consensus state: current_term: 1 leader_uuid: "61972f53d4b34f2d9d834eb231892574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } } }
I20260810 04:18:46.549993 30952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:46.550258 30951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61972f53d4b34f2d9d834eb231892574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } } }
I20260810 04:18:46.550367 30951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:46.550530 30961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:18:46.551090 30961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:18:46.553324 30961 catalog_manager.cc:1347] Generated new cluster ID: 39ddaeab0a734db8947177d580543c7e
I20260810 04:18:46.553385 30961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:18:46 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:46.560688 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:46.549817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.15.216.126:32953)
0810 04:18:46.550013 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:46.550023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:46.550061 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:18:46.550195 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:46.550204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:46.550264 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:46.551296 (+ 1032us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:46.551418 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.552392 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.553326 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.553894 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.554018 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:46.556426 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:46.556431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:46.556431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:46.556452 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:46.558953 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:46.559352 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:46.559356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:46.559356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:46.559483 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:46.559721 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:46.559734 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:46.559735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:46.560007 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:46.560194 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:46.560308 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 04:18:46.560471 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260810 04:18:46.575753 30961 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:18:46.576990 30961 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:18:46.581068 30961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Generated new TSK 0
I20260810 04:18:46.581310 30961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:18:46.615098 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:32953
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:46.716212 30973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:46.716421 30973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:46.716446 30973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:46.717507 30973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:46.717541 30973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:46.717567 30973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:18:46.719235 30973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:46.719431 30973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:46.719646 30973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:46.725064 30982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:46.725073 30980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:46.725148 30973 server_base.cc:1034] running on GCE node
W20260810 04:18:46.725060 30979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:46.725651 30973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:46.725944 30973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:46.727125 30973 hybrid_clock.cc:648] HybridClock initialized: now 1786335526727110 us; error 26 us; skew 500 ppm
Aug 10 04:18:46 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:46.729934 30973 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:18:46.730407 30973 webserver.cc:460] Webserver started at http://127.15.216.65:39795/ using document root <none> and password file <none>
I20260810 04:18:46.730672 30973 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:46.730744 30973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:46.730904 30973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:46.731977 30973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "9b2196167e6a9ac144b120cad31fb128"
server_key_iv: "0b4a699f35c9769c5836f65a71eaf886"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.732342 30973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "9b2196167e6a9ac144b120cad31fb128"
server_key_iv: "0b4a699f35c9769c5836f65a71eaf886"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.733726 30973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 04:18:46.734592 30989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:46.734825 30973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:46.734949 30973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "9b2196167e6a9ac144b120cad31fb128"
server_key_iv: "0b4a699f35c9769c5836f65a71eaf886"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.735038 30973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:46.742833 30973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:46.744892 30973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:46.745055 30973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:46.745294 30973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:46.745705 30973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:46.745759 30973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:46.745805 30973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:46.745831 30973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:46.752305 30973 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:45891
I20260810 04:18:46.752382 31117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:45891 every 8 connection(s)
I20260810 04:18:46.752754 30973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:18:46 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:46.762125 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 30973
I20260810 04:18:46.762264 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:18:46.762457 16225 external_mini_cluster.cc:1428] Setting key b10bbc3c5440b0eb6e9b0ae0f9359b02
I20260810 04:18:46.763978 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:46.753763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:50913 (local address 127.15.216.126:32953)
0810 04:18:46.753867 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:46.753869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:46.754241 (+ 372us) server_negotiation.cc:407] Connection header received
0810 04:18:46.754793 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:46.754795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:46.754810 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:46.754873 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:46.755894 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.756435 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.757504 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.757817 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.757882 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:46.760349 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:46.760351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:46.760351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:46.760358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:46.762013 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:46.762678 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:46.762680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:46.762681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:46.762764 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:46.763118 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:46.763119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:46.763119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:46.763279 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:46.763391 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:46.763823 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 04:18:46.763911 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 04:18:46.764282 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:46.753780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:50913)
0810 04:18:46.754030 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:46.754071 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:46.754668 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:46.754938 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:46.754942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:46.755577 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:46.755776 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:46.755780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.756538 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.756539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:46.757408 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:46.757411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.757949 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.757950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:46.758083 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:46.758747 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:46.758755 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:46.760234 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:46.762120 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:46.762130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:46.762133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:46.762526 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:46.762862 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:46.762864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:46.762864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:46.763027 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:46.763403 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:46.763410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:46.763611 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 04:18:46.763751 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 04:18:46.763951 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260810 04:18:46.764941 31118 heartbeater.cc:347] Connected to a master server at 127.15.216.126:32953
I20260810 04:18:46.765041 31118 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:46.765313 31118 heartbeater.cc:511] Master 127.15.216.126:32953 requested a full tablet report, sending...
I20260810 04:18:46.765947 30906 ts_manager.cc:194] Registered new tserver with Master: 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891)
I20260810 04:18:46.766654 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:50913
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:18:46.792857 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:32953
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:46.878566 31123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:46.878762 31123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:46.878785 31123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:46.879905 31123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:46.879947 31123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:46.880003 31123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:18:46.881477 31123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:46.881749 31123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:46.881983 31123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:46.887157 31131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:46.887463 31128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:46.887604 31123 server_base.cc:1034] running on GCE node
W20260810 04:18:46.887728 31129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:46.887969 31123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:46.888173 31123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:46.889345 31123 hybrid_clock.cc:648] HybridClock initialized: now 1786335526889327 us; error 33 us; skew 500 ppm
Aug 10 04:18:46 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:46.891969 31123 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:18:46.892477 31123 webserver.cc:460] Webserver started at http://127.15.216.66:36767/ using document root <none> and password file <none>
I20260810 04:18:46.892709 31123 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:46.892779 31123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:46.892926 31123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:46.894075 31123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "7ff79c3242b04e33a80673f792c0332b"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "ddb7f1fbb572d95ce162b573da7377ba"
server_key_iv: "96bc8bdd84fc6cb94e11e7cba270156d"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.894438 31123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "7ff79c3242b04e33a80673f792c0332b"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "ddb7f1fbb572d95ce162b573da7377ba"
server_key_iv: "96bc8bdd84fc6cb94e11e7cba270156d"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.895789 31123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:18:46.896572 31138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:46.896761 31123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:46.896875 31123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "7ff79c3242b04e33a80673f792c0332b"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "ddb7f1fbb572d95ce162b573da7377ba"
server_key_iv: "96bc8bdd84fc6cb94e11e7cba270156d"
server_key_version: "encryptionkey@0"
I20260810 04:18:46.896951 31123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:46.920503 31123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:46.922986 31123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:46.923141 31123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:46.923404 31123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:46.923735 31123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:46.923772 31123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:46.923796 31123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:46.923808 31123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:46.930182 31123 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:36765
I20260810 04:18:46.930255 31259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:36765 every 8 connection(s)
I20260810 04:18:46.930584 31123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:18:46 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335526, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:46.939227 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 31123
I20260810 04:18:46.939380 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:18:46.939580 16225 external_mini_cluster.cc:1428] Setting key f79ddbd19f58f376cb489f59f0595d90
I20260810 04:18:46.939991 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:46.931374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:46681 (local address 127.15.216.126:32953)
0810 04:18:46.931446 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:46.931449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:46.931657 (+ 208us) server_negotiation.cc:407] Connection header received
0810 04:18:46.932155 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:46.932156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:46.932167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:46.932220 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:46.933151 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.933719 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.934668 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.934870 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.934926 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:46.937123 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:46.937125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:46.937125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:46.937131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:46.938511 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:46.938895 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:46.938896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:46.938896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:46.938979 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:46.939284 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:46.939285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:46.939285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:46.939445 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:46.939507 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:46.939853 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 04:18:46.939927 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:18:46.940199 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:46.931339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:46681)
0810 04:18:46.931535 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:46.931573 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:46.932070 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:46.932270 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:46.932273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:46.932846 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:46.933062 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:46.933066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.933808 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.933808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:46.934590 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:46.934592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:46.934948 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:46.934949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:46.935071 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:46.935657 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:46.935666 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:46.937035 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:46.938584 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:46.938591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:46.938593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:46.938803 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:46.939056 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:46.939057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:46.939058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:46.939199 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:46.939510 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:46.939518 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:46.939676 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 04:18:46.939795 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 04:18:46.939973 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260810 04:18:46.940833 31260 heartbeater.cc:347] Connected to a master server at 127.15.216.126:32953
I20260810 04:18:46.940938 31260 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:46.941126 31260 heartbeater.cc:511] Master 127.15.216.126:32953 requested a full tablet report, sending...
I20260810 04:18:46.941565 30906 ts_manager.cc:194] Registered new tserver with Master: 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765)
I20260810 04:18:46.942009 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:46681
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:18:46.969492 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:32953
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:47.057448 31265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:47.057717 31265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:47.057755 31265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:47.058902 31265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:47.058943 31265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:47.058969 31265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:18:47.060425 31265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:47.060674 31265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:47.060907 31265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:47.066233 31270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.066370 31265 server_base.cc:1034] running on GCE node
W20260810 04:18:47.066286 31273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:47.066293 31271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.066700 31265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:47.066912 31265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:47.068058 31265 hybrid_clock.cc:648] HybridClock initialized: now 1786335527068031 us; error 40 us; skew 500 ppm
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:47.070681 31265 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:18:47.071105 31265 webserver.cc:460] Webserver started at http://127.15.216.67:42001/ using document root <none> and password file <none>
I20260810 04:18:47.071331 31265 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:47.071408 31265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:47.071563 31265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:18:47.072590 31265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "3d4cc3000d434582b529d498123b5605"
format_stamp: "Formatted at 2026-08-10 04:18:47 on dist-test-slave-g350"
server_key: "6d3b3f20719fcdf58622de7e551da229"
server_key_iv: "07df2c4a9c21db3905dd41534e0cba9f"
server_key_version: "encryptionkey@0"
I20260810 04:18:47.072955 31265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "3d4cc3000d434582b529d498123b5605"
format_stamp: "Formatted at 2026-08-10 04:18:47 on dist-test-slave-g350"
server_key: "6d3b3f20719fcdf58622de7e551da229"
server_key_iv: "07df2c4a9c21db3905dd41534e0cba9f"
server_key_version: "encryptionkey@0"
I20260810 04:18:47.074481 31265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:18:47.075296 31280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.075488 31265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:47.075598 31265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "3d4cc3000d434582b529d498123b5605"
format_stamp: "Formatted at 2026-08-10 04:18:47 on dist-test-slave-g350"
server_key: "6d3b3f20719fcdf58622de7e551da229"
server_key_iv: "07df2c4a9c21db3905dd41534e0cba9f"
server_key_version: "encryptionkey@0"
I20260810 04:18:47.075682 31265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:47.088261 31265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:47.095353 31265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:47.095526 31265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:47.095778 31265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:47.096138 31265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:47.096196 31265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.096261 31265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:47.096292 31265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.102874 31265 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:38545
I20260810 04:18:47.102942 31401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:38545 every 8 connection(s)
I20260810 04:18:47.103276 31265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:18:47.105366 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 31265
I20260810 04:18:47.105443 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:18:47.105672 16225 external_mini_cluster.cc:1428] Setting key 4711150a5bb5e7dfac08f4547f378803
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:47.113534 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.104082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:52155 (local address 127.15.216.126:32953)
0810 04:18:47.104153 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.104155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.104380 (+ 225us) server_negotiation.cc:407] Connection header received
0810 04:18:47.104878 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.104880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.104893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.104964 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:47.106287 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.106946 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.107986 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.108192 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.108258 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.110502 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:47.110504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:47.110504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:47.110511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:47.112306 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.112686 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.112687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.112687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.112741 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.112973 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.112974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.112974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.113084 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:47.113136 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.113401 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.113470 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:18:47.113734 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.104031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:52155)
0810 04:18:47.104202 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.104247 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.104757 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.105044 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.105047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.105903 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:47.106206 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.106213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.107049 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.107050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.107900 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.107902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.108275 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.108277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.108381 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.108857 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:47.108865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:47.110407 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:47.112387 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.112393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.112395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.112612 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.112810 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.112810 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.112811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.112913 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.113143 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:47.113149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:47.113261 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.113362 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.113482 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260810 04:18:47.114270 31402 heartbeater.cc:347] Connected to a master server at 127.15.216.126:32953
I20260810 04:18:47.114358 31402 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:47.114497 31402 heartbeater.cc:511] Master 127.15.216.126:32953 requested a full tablet report, sending...
I20260810 04:18:47.114811 30906 ts_manager.cc:194] Registered new tserver with Master: 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545)
I20260810 04:18:47.115139 30906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:52155
I20260810 04:18:47.117758 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:18:47.125435 30956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.119039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.15.216.126:32953)
0810 04:18:47.119115 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.119117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.119237 (+ 120us) server_negotiation.cc:407] Connection header received
0810 04:18:47.119316 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.119317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.119327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.119426 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:47.120006 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.120488 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.121227 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.121426 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.121502 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.122746 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:47.122747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:47.122748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:47.122754 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:47.124212 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.124579 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.124580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.124580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.124640 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.124852 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.124853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.124854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.125004 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:47.125112 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.125247 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.125326 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 04:18:47.127180 30906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:18:47.128810 30906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:18:47.135476 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.131242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:36765 (local address 127.0.0.1:36614)
0810 04:18:47.131411 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.131455 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.131522 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.131966 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.131966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.131968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.132036 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.132039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.133092 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.133093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.134301 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.134304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.135130 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.135131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.135212 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.135226 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.135332 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.135389 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260810 04:18:47.135485 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.131350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.15.216.66:36765)
0810 04:18:47.131774 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.131777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.131809 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:18:47.131852 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.131854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.131868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.131941 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.132133 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.132987 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.134390 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.135039 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.135128 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.135191 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.135353 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.135398 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":134,"threads_started":1}
I20260810 04:18:47.135814 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.131276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:38545 (local address 127.0.0.1:43848)
0810 04:18:47.132050 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.132061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.132112 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.132329 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.132330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.132332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.132372 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.132373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.133604 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.133605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.134667 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.134670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.135587 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.135588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.135637 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.135640 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.135691 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.135737 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260810 04:18:47.135932 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.131540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.15.216.65:45891)
0810 04:18:47.131739 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.131744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.131756 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:18:47.131799 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.131801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.131820 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.131912 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.132239 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.133397 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.134618 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.135332 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.135419 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.135476 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.135785 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.135828 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260810 04:18:47.136106 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.131275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:45891 (local address 127.0.0.1:54362)
0810 04:18:47.131577 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.131588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.131629 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.131957 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.131960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.131962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.132036 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.132039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.133508 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.133509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.134535 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.134536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.135515 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.135516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.135634 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.135638 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.135691 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.135739 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260810 04:18:47.136137 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.131507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.15.216.67:38545)
0810 04:18:47.131708 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.131713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.132175 (+ 462us) server_negotiation.cc:407] Connection header received
0810 04:18:47.132220 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.132222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.132238 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.132317 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.132460 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.133504 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.134753 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.135440 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.135503 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.135552 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.136004 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.136056 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":118,"threads_started":1}
I20260810 04:18:47.136443 31189 tablet_service.cc:1467] Processing CreateTablet for tablet b39420fdee66435a8770cbe77c2aab00 (DEFAULT_TABLE table=test-table [id=c6f96bc895064fb3b50bc078deab96cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:47.136823 31189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39420fdee66435a8770cbe77c2aab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:47.136822 31331 tablet_service.cc:1467] Processing CreateTablet for tablet b39420fdee66435a8770cbe77c2aab00 (DEFAULT_TABLE table=test-table [id=c6f96bc895064fb3b50bc078deab96cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:47.136816 31039 tablet_service.cc:1467] Processing CreateTablet for tablet b39420fdee66435a8770cbe77c2aab00 (DEFAULT_TABLE table=test-table [id=c6f96bc895064fb3b50bc078deab96cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:47.137144 31331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39420fdee66435a8770cbe77c2aab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:47.137144 31039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b39420fdee66435a8770cbe77c2aab00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:47.139725 31421 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Bootstrap starting.
I20260810 04:18:47.140442 31422 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Bootstrap starting.
I20260810 04:18:47.140535 31421 tablet_bootstrap.cc:654] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:47.140689 31423 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap starting.
I20260810 04:18:47.141163 31421 log.cc:824] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:47.141188 31422 tablet_bootstrap.cc:654] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:47.141337 31423 tablet_bootstrap.cc:654] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:47.141815 31422 log.cc:824] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:47.141988 31423 log.cc:824] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:47.142331 31421 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: No bootstrap required, opened a new log
I20260810 04:18:47.142462 31421 ts_tablet_manager.cc:1397] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:18:47.142714 31422 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: No bootstrap required, opened a new log
I20260810 04:18:47.142797 31422 ts_tablet_manager.cc:1397] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:18:47.143137 31423 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: No bootstrap required, opened a new log
I20260810 04:18:47.143214 31423 ts_tablet_manager.cc:1397] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:18:47.144237 31422 raft_consensus.cc:348] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.144410 31422 raft_consensus.cc:374] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:47.144459 31422 raft_consensus.cc:729] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4cc3000d434582b529d498123b5605, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.144570 31422 consensus_queue.cc:260] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.144569 31421 raft_consensus.cc:348] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.144724 31421 raft_consensus.cc:374] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:47.144755 31421 raft_consensus.cc:729] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff79c3242b04e33a80673f792c0332b, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.144879 31422 ts_tablet_manager.cc:1428] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:18:47.144893 31402 heartbeater.cc:503] Master 127.15.216.126:32953 was elected leader, sending a full tablet report...
I20260810 04:18:47.144909 31421 consensus_queue.cc:260] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.145190 31421 ts_tablet_manager.cc:1428] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:18:47.145251 31260 heartbeater.cc:503] Master 127.15.216.126:32953 was elected leader, sending a full tablet report...
I20260810 04:18:47.145298 31423 raft_consensus.cc:348] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.145469 31423 raft_consensus.cc:374] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:47.145531 31423 raft_consensus.cc:729] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd7b7bc9d9c4f2a8dca9ae08117474f, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.145678 31423 consensus_queue.cc:260] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.146144 31118 heartbeater.cc:503] Master 127.15.216.126:32953 was elected leader, sending a full tablet report...
I20260810 04:18:47.146258 31423 ts_tablet_manager.cc:1428] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:18:47.165236 31428 raft_consensus.cc:482] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:47.165370 31428 raft_consensus.cc:504] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.165868 31428 leader_election.cc:288] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545), 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891)
I20260810 04:18:47.169087 31262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.166060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:38545 (local address 127.15.216.66:53557)
0810 04:18:47.166120 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.166133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.166177 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.166393 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.166394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.166397 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.166505 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.166506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.167168 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.167169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.168054 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.168056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.168782 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.168784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.168888 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.168892 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.168958 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.169003 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":2}
I20260810 04:18:47.169116 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.166136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:53557 (local address 127.15.216.67:38545)
0810 04:18:47.166186 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.166189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.166227 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:18:47.166259 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.166260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.166271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.166353 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.166586 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.167088 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.168164 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.168700 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.168768 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.168775 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.168996 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.169038 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:18:47.169409 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.166140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56433 (local address 127.15.216.65:45891)
0810 04:18:47.166186 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.166189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.166401 (+ 212us) server_negotiation.cc:407] Connection header received
0810 04:18:47.166432 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.166433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.166444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.166497 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.166674 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.167292 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.168474 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.168928 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.168994 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.169002 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.169214 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.169334 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:18:47.169481 31352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39420fdee66435a8770cbe77c2aab00" candidate_uuid: "7ff79c3242b04e33a80673f792c0332b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4cc3000d434582b529d498123b5605" is_pre_election: true
I20260810 04:18:47.169700 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.166060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:45891 (local address 127.15.216.66:56433)
0810 04:18:47.166264 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.166275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.166356 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.166510 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.166511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.166512 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.166601 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.166603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.167409 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.167410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.168375 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.168378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.169011 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.169012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.169099 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.169103 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.169527 (+ 424us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.169581 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":16,"thread_start_us":138,"threads_started":1}
I20260810 04:18:47.169754 31352 raft_consensus.cc:2454] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff79c3242b04e33a80673f792c0332b in term 0.
I20260810 04:18:47.170050 31142 leader_election.cc:302] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b; no voters:
I20260810 04:18:47.170053 31061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39420fdee66435a8770cbe77c2aab00" candidate_uuid: "7ff79c3242b04e33a80673f792c0332b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" is_pre_election: true
I20260810 04:18:47.170189 31428 raft_consensus.cc:2781] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:18:47.170228 31061 raft_consensus.cc:2454] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff79c3242b04e33a80673f792c0332b in term 0.
I20260810 04:18:47.170256 31428 raft_consensus.cc:482] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:18:47.170289 31428 raft_consensus.cc:3037] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:47.171165 31428 raft_consensus.cc:504] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.171365 31428 leader_election.cc:288] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [CANDIDATE]: Term 1 election: Requested vote from peers 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545), 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891)
I20260810 04:18:47.171528 31352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39420fdee66435a8770cbe77c2aab00" candidate_uuid: "7ff79c3242b04e33a80673f792c0332b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4cc3000d434582b529d498123b5605"
I20260810 04:18:47.171634 31352 raft_consensus.cc:3037] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:47.171619 31061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b39420fdee66435a8770cbe77c2aab00" candidate_uuid: "7ff79c3242b04e33a80673f792c0332b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f"
I20260810 04:18:47.171720 31061 raft_consensus.cc:3037] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:47.172420 31352 raft_consensus.cc:2454] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff79c3242b04e33a80673f792c0332b in term 1.
I20260810 04:18:47.172559 31061 raft_consensus.cc:2454] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff79c3242b04e33a80673f792c0332b in term 1.
I20260810 04:18:47.172665 31142 leader_election.cc:302] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b; no voters:
I20260810 04:18:47.172792 31428 raft_consensus.cc:2781] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:18:47.173014 31428 raft_consensus.cc:686] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 1 LEADER]: Becoming Leader. State: Replica: 7ff79c3242b04e33a80673f792c0332b, State: Running, Role: LEADER
I20260810 04:18:47.173143 31428 consensus_queue.cc:237] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.174110 30906 catalog_manager.cc:5495] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff79c3242b04e33a80673f792c0332b (127.15.216.66). New cstate: current_term: 1 leader_uuid: "7ff79c3242b04e33a80673f792c0332b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } health_report { overall_health: UNKNOWN } } }
W20260810 04:18:47.181146 31261 tablet.cc:2369] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:18:47.184514 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.182270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.15.216.66:36765)
0810 04:18:47.182328 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.182330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.182339 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:18:47.182382 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.182383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.182399 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.182458 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:18:47.182591 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.183054 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.183841 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.184034 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.184090 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.184238 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:18:47.184327 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:18:47.184388 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.184428 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.184445 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:18:47.186362 31352 raft_consensus.cc:1261] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 1 FOLLOWER]: Refusing update from remote peer 7ff79c3242b04e33a80673f792c0332b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:47.186393 31061 raft_consensus.cc:1261] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Refusing update from remote peer 7ff79c3242b04e33a80673f792c0332b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:47.186780 31431 consensus_queue.cc:1035] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:47.186933 31428 consensus_queue.cc:1035] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:47.190083 31435 mvcc.cc:205] Tried to move back new op lower bound from 7316830319352168448 to 7316830319302053888. Current Snapshot: MvccSnapshot[applied={T|T < 7316830319352168448 or (T in {7316830319352168448})}]
I20260810 04:18:47.191532 30905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:18:47.191613 30905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:18:47.192368 30905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Sending DeleteTablet for 3 replicas of tablet b39420fdee66435a8770cbe77c2aab00
I20260810 04:18:47.192672 31189 tablet_service.cc:1514] Processing DeleteTablet for tablet b39420fdee66435a8770cbe77c2aab00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:47 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:36614
I20260810 04:18:47.192680 31331 tablet_service.cc:1514] Processing DeleteTablet for tablet b39420fdee66435a8770cbe77c2aab00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:47 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43848
I20260810 04:18:47.192698 31039 tablet_service.cc:1514] Processing DeleteTablet for tablet b39420fdee66435a8770cbe77c2aab00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:47 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:54362
I20260810 04:18:47.192930 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30973
I20260810 04:18:47.192940 31444 tablet_replica.cc:330] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: stopping tablet replica
I20260810 04:18:47.192936 31446 tablet_replica.cc:330] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: stopping tablet replica
I20260810 04:18:47.193033 31444 raft_consensus.cc:2229] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:18:47.193037 31446 raft_consensus.cc:2229] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:18:47.193135 31444 raft_consensus.cc:2258] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:18:47.193140 31446 raft_consensus.cc:2258] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:18:47.194409 31444 ts_tablet_manager.cc:1905] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:18:47.194450 31446 ts_tablet_manager.cc:1905] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:18:47.196316 31444 ts_tablet_manager.cc:1918] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:18:47.196416 31444 log.cc:1196] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/b39420fdee66435a8770cbe77c2aab00
I20260810 04:18:47.196674 31446 ts_tablet_manager.cc:1918] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:18:47.196749 31444 ts_tablet_manager.cc:1939] T b39420fdee66435a8770cbe77c2aab00 P 7ff79c3242b04e33a80673f792c0332b: Deleting consensus metadata
I20260810 04:18:47.196779 31446 log.cc:1196] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/b39420fdee66435a8770cbe77c2aab00
I20260810 04:18:47.197059 31446 ts_tablet_manager.cc:1939] T b39420fdee66435a8770cbe77c2aab00 P 3d4cc3000d434582b529d498123b5605: Deleting consensus metadata
I20260810 04:18:47.197209 30890 catalog_manager.cc:4841] TS 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765): tablet b39420fdee66435a8770cbe77c2aab00 (table test-table [id=c6f96bc895064fb3b50bc078deab96cb]) successfully deleted
I20260810 04:18:47.197464 30891 catalog_manager.cc:4841] TS 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545): tablet b39420fdee66435a8770cbe77c2aab00 (table test-table [id=c6f96bc895064fb3b50bc078deab96cb]) successfully deleted
W20260810 04:18:47.202781 30889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:45891 (error 108)
I20260810 04:18:47.203331 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.203208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:45891 (local address 127.0.0.1:54376)
0810 04:18:47.203251 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.203295 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260810 04:18:47.203326 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 31123
W20260810 04:18:47.206722 30889 catalog_manager.cc:4584] TS 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b39420fdee66435a8770cbe77c2aab00: Network error: Client connection negotiation failed: client connection to 127.15.216.65:45891: connect: Connection refused (error 111)
I20260810 04:18:47.211198 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 31265
I20260810 04:18:47.218209 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 30872
I20260810 04:18:47.225926 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:32953
--webserver_interface=127.15.216.126
--webserver_port=40213
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:47.312927 31448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:47.313135 31448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:47.313159 31448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:47.314394 31448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:47.314436 31448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:47.314451 31448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:47.314463 31448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:47.314476 31448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:47.315976 31448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:32953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:47.316244 31448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:47.316506 31448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:47.326771 31457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:47.326758 31456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:47.326758 31459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.327073 31448 server_base.cc:1034] running on GCE node
I20260810 04:18:47.327375 31448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:47.327739 31448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:47.328913 31448 hybrid_clock.cc:648] HybridClock initialized: now 1786335527328893 us; error 36 us; skew 500 ppm
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:47.331606 31448 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:18:47.332127 31448 webserver.cc:460] Webserver started at http://127.15.216.126:40213/ using document root <none> and password file <none>
I20260810 04:18:47.332396 31448 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:47.332489 31448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:47.334039 31448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:18:47.334856 31468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.335067 31448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:18:47.335186 31448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "61972f53d4b34f2d9d834eb231892574"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "bb06b5e29f340246f54f9de303d0e1ed"
server_key_iv: "62d98d8430c7896610bd27f7df42bc06"
server_key_version: "encryptionkey@0"
I20260810 04:18:47.335624 31448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:47.346848 31448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:47.349035 31448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:47.349222 31448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:47.353735 31448 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:32953
I20260810 04:18:47.353801 31523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:32953 every 8 connection(s)
I20260810 04:18:47.354225 31448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:18:47.356027 31524 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:18:47.356739 31524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Bootstrap starting.
I20260810 04:18:47.361923 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 31448
I20260810 04:18:47.362242 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:45891
--local_ip_for_outbound_sockets=127.15.216.65
--tserver_master_addrs=127.15.216.126:32953
--webserver_port=39795
--webserver_interface=127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:47.366315 31524 log.cc:824] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:47.368312 31524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:18:47.368635 31524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Bootstrap complete.
I20260810 04:18:47.371096 31524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:47.371357 31524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61972f53d4b34f2d9d834eb231892574, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.371542 31524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:47.371623 31524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:18:47.371683 31524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:18:47.371731 31524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:18:47.372763 31524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:47.372889 31524 leader_election.cc:302] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61972f53d4b34f2d9d834eb231892574; no voters:
I20260810 04:18:47.373083 31524 leader_election.cc:288] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:18:47.373152 31528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:18:47.373298 31528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 LEADER]: Becoming Leader. State: Replica: 61972f53d4b34f2d9d834eb231892574, State: Running, Role: LEADER
I20260810 04:18:47.373375 31524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:47.373476 31528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:47.373777 31528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61972f53d4b34f2d9d834eb231892574. Latest consensus state: current_term: 2 leader_uuid: "61972f53d4b34f2d9d834eb231892574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } } }
I20260810 04:18:47.373847 31528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:47.373812 31529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "61972f53d4b34f2d9d834eb231892574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } } }
I20260810 04:18:47.374065 31529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:47.374069 31534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:18:47.374894 31534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:18:47.374989 31534 catalog_manager.cc:1259] Loaded cluster ID: 39ddaeab0a734db8947177d580543c7e
I20260810 04:18:47.375012 31534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:18:47.375252 31534 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 04:18:47.380362 31534 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 04:18:47.383574 31534 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:18:47.383780 31534 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Loaded TSK: 0
I20260810 04:18:47.383996 31534 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:18:47.450381 31526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:47.450580 31526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:47.450603 31526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:47.451731 31526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:47.451773 31526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:47.451799 31526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:18:47.453240 31526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:45891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=39795
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:47.453485 31526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:47.453750 31526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:47.460075 31553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:47.460074 31555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.460425 31526 server_base.cc:1034] running on GCE node
W20260810 04:18:47.460116 31551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.460698 31526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:47.460954 31526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:47.462121 31526 hybrid_clock.cc:648] HybridClock initialized: now 1786335527462107 us; error 32 us; skew 500 ppm
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:47.464825 31526 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:18:47.465291 31526 webserver.cc:460] Webserver started at http://127.15.216.65:39795/ using document root <none> and password file <none>
I20260810 04:18:47.465552 31526 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:47.465675 31526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:47.467144 31526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:18:47.467914 31562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.468128 31526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:18:47.468238 31526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "9b2196167e6a9ac144b120cad31fb128"
server_key_iv: "0b4a699f35c9769c5836f65a71eaf886"
server_key_version: "encryptionkey@0"
I20260810 04:18:47.468704 31526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:47.484170 31526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:47.486776 31526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:47.486958 31526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:47.487226 31526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:47.487757 31571 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:18:47.489450 31526 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:18:47.489514 31526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:18:47.489549 31526 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:18:47.490434 31526 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:18:47.490482 31526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:18:47.490563 31571 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap starting.
I20260810 04:18:47.497438 31526 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:45891
I20260810 04:18:47.497768 31688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:45891 every 8 connection(s)
I20260810 04:18:47.497918 31526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:18:47.498454 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 31526
I20260810 04:18:47.498792 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:36765
--local_ip_for_outbound_sockets=127.15.216.66
--tserver_master_addrs=127.15.216.126:32953
--webserver_port=36767
--webserver_interface=127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:47.502730 31571 log.cc:824] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:47.505672 31571 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:18:47.506031 31571 tablet_bootstrap.cc:492] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap complete.
I20260810 04:18:47.507527 31571 ts_tablet_manager.cc:1397] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:47.509733 31571 raft_consensus.cc:348] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.510139 31571 raft_consensus.cc:729] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd7b7bc9d9c4f2a8dca9ae08117474f, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.510316 31571 consensus_queue.cc:260] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.510634 31571 ts_tablet_manager.cc:1428] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 04:18:47.515113 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.501379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:43515)
0810 04:18:47.501777 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.501827 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.502544 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.503214 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.503219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.503577 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:47.503862 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.503870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.505627 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.505628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.507009 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.507012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.507950 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.507952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.508092 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.508746 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:47.508764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:47.510761 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:47.513302 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.513310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.513312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.513620 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.513910 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.513911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.513911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.514044 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.514490 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:47.514497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:47.514644 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.514741 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.514915 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":128,"threads_started":1}
I20260810 04:18:47.515640 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.501460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:43515 (local address 127.15.216.126:32953)
0810 04:18:47.502174 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.502184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.502230 (+ 46us) server_negotiation.cc:407] Connection header received
0810 04:18:47.502790 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.502798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.502927 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.503069 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:47.504033 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.505440 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.507108 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.507736 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.507843 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.510893 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:47.510897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:47.510897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:47.510910 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:47.513199 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.513702 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.513706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.513707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.513825 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.514124 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.514125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.514126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.514412 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:47.514494 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.515315 (+ 821us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.515447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":113,"threads_started":1}
I20260810 04:18:47.516276 31689 heartbeater.cc:347] Connected to a master server at 127.15.216.126:32953
I20260810 04:18:47.516393 31689 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:47.516575 31689 heartbeater.cc:511] Master 127.15.216.126:32953 requested a full tablet report, sending...
I20260810 04:18:47.517282 31486 ts_manager.cc:194] Registered new tserver with Master: 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891)
I20260810 04:18:47.518421 31486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:43515
I20260810 04:18:47.519392 31689 heartbeater.cc:503] Master 127.15.216.126:32953 was elected leader, sending a full tablet report...
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:18:47.526736 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.517924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:45891 (local address 127.0.0.1:54388)
0810 04:18:47.518091 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.518126 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.518193 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.518472 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.518477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.518803 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:47.518845 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.518847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.519888 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.519889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.520838 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.520841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.521290 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.521291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.521369 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.521874 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:47.521879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:47.523284 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:47.525410 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.525419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.525420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.525793 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.526028 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.526029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.526029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.526090 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.526436 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:47.526443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:47.526490 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.526566 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.526667 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260810 04:18:47.526871 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.518165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.15.216.65:45891)
0810 04:18:47.518311 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.518315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.518325 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:18:47.518363 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.518365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.518379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.518455 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:47.518934 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.519822 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.520947 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.521213 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.521268 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.523376 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:47.523385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:47.523385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:47.523392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:47.525300 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.525883 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.525886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.525886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.525954 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.526156 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.526157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.526157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.526344 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:47.526431 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.526625 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.526732 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260810 04:18:47.527333 31615 tablet_service.cc:1514] Processing DeleteTablet for tablet b39420fdee66435a8770cbe77c2aab00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:47 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:54388
I20260810 04:18:47.527894 31698 tablet_replica.cc:330] stopping tablet replica
I20260810 04:18:47.528005 31698 raft_consensus.cc:2229] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:18:47.528074 31698 raft_consensus.cc:2258] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:18:47.528390 31698 ts_tablet_manager.cc:1905] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:18:47.530880 31698 ts_tablet_manager.cc:1918] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:18:47.530962 31698 log.cc:1196] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/b39420fdee66435a8770cbe77c2aab00
I20260810 04:18:47.531193 31698 ts_tablet_manager.cc:1939] T b39420fdee66435a8770cbe77c2aab00 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Deleting consensus metadata
I20260810 04:18:47.531821 31470 catalog_manager.cc:4841] TS 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891): tablet b39420fdee66435a8770cbe77c2aab00 (table test-table [id=c6f96bc895064fb3b50bc078deab96cb]) successfully deleted
W20260810 04:18:47.594954 31691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:47.595152 31691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:47.595175 31691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:47.596343 31691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:47.596387 31691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:47.596414 31691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:18:47.597957 31691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:36765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=36767
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:47.598201 31691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:47.598434 31691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:47.603770 31703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:47.603855 31706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:47.603892 31704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.604106 31691 server_base.cc:1034] running on GCE node
I20260810 04:18:47.604314 31691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:47.604547 31691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:47.605671 31691 hybrid_clock.cc:648] HybridClock initialized: now 1786335527605665 us; error 36 us; skew 500 ppm
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:47.608289 31691 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:18:47.608672 31691 webserver.cc:460] Webserver started at http://127.15.216.66:36767/ using document root <none> and password file <none>
I20260810 04:18:47.608867 31691 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:47.608923 31691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:47.610289 31691 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:47.610883 31715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.611068 31691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:47.611141 31691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "7ff79c3242b04e33a80673f792c0332b"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "ddb7f1fbb572d95ce162b573da7377ba"
server_key_iv: "96bc8bdd84fc6cb94e11e7cba270156d"
server_key_version: "encryptionkey@0"
I20260810 04:18:47.611441 31691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:47.634873 31691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:47.637231 31691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:47.637349 31691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:47.637558 31691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:47.637920 31691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:47.637955 31691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.637979 31691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:47.638026 31691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.643977 31691 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:36765
I20260810 04:18:47.644134 31839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:36765 every 8 connection(s)
I20260810 04:18:47.644445 31691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:18:47.646890 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 31691
I20260810 04:18:47.647192 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:38545
--local_ip_for_outbound_sockets=127.15.216.67
--tserver_master_addrs=127.15.216.126:32953
--webserver_port=42001
--webserver_interface=127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:47.655133 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.645205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:41879 (local address 127.15.216.126:32953)
0810 04:18:47.645279 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.645286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.645516 (+ 230us) server_negotiation.cc:407] Connection header received
0810 04:18:47.646086 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.646088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.646101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.646165 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:47.647396 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.648161 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.649507 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.649818 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.649904 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.652143 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:47.652145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:47.652145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:47.652154 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:47.653678 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.654112 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.654114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.654114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.654180 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.654450 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.654451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.654452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.654575 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:47.654651 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.654943 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.655038 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 04:18:47.655339 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.645208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:41879)
0810 04:18:47.645389 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.645426 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.645968 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.646240 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.646243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.646968 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:47.647270 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.647277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.648287 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.648288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.649411 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.649414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.649920 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.649921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.650049 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.650597 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:47.650606 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:47.652039 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:47.653777 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.653792 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.653794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.654028 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.654248 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.654249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.654249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.654367 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.654641 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:47.654649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:47.654801 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.654910 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.655079 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260810 04:18:47.655916 31840 heartbeater.cc:347] Connected to a master server at 127.15.216.126:32953
I20260810 04:18:47.656045 31840 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:47.656275 31840 heartbeater.cc:511] Master 127.15.216.126:32953 requested a full tablet report, sending...
I20260810 04:18:47.656702 31486 ts_manager.cc:194] Registered new tserver with Master: 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765)
I20260810 04:18:47.657124 31486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:41879
W20260810 04:18:47.732183 31843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:47.732388 31843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:47.732412 31843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:47.733546 31843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:47.733637 31843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:47.733708 31843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:18:47.735221 31843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:38545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=42001
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:47.735473 31843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:47.735713 31843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:47.741046 31848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:47.741053 31851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.741180 31843 server_base.cc:1034] running on GCE node
W20260810 04:18:47.741145 31849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:47.741644 31843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:47.741873 31843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:47.743018 31843 hybrid_clock.cc:648] HybridClock initialized: now 1786335527743008 us; error 27 us; skew 500 ppm
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:47.745785 31843 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:18:47.746263 31843 webserver.cc:460] Webserver started at http://127.15.216.67:42001/ using document root <none> and password file <none>
I20260810 04:18:47.746500 31843 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:47.746585 31843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:47.747972 31843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:47.748685 31858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.748905 31843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:47.749010 31843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "3d4cc3000d434582b529d498123b5605"
format_stamp: "Formatted at 2026-08-10 04:18:47 on dist-test-slave-g350"
server_key: "6d3b3f20719fcdf58622de7e551da229"
server_key_iv: "07df2c4a9c21db3905dd41534e0cba9f"
server_key_version: "encryptionkey@0"
I20260810 04:18:47.749367 31843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:47.758934 31843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:47.761153 31843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:47.761337 31843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:47.761579 31843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:47.761976 31843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:47.762033 31843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.762076 31843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:47.762104 31843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:47.768339 31843 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:38545
I20260810 04:18:47.768392 31978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:38545 every 8 connection(s)
I20260810 04:18:47.768729 31843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:18:47.773154 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 31843
Aug 10 04:18:47 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335527, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:18:47.779246 31693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.769489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:60115 (local address 127.15.216.126:32953)
0810 04:18:47.769574 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.769576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.769798 (+ 222us) server_negotiation.cc:407] Connection header received
0810 04:18:47.770382 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.770384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.770396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.770469 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:47.771429 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.771935 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.773049 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.773354 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.773465 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.775993 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:47.775995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:47.775995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:47.776004 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:47.777782 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.778208 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.778210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.778211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.778278 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.778536 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.778537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.778537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.778657 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:47.778737 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.779017 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.779155 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 04:18:47.779400 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.769492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:60115)
0810 04:18:47.769667 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.769713 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.770242 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.770523 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.770526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.771110 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:47.771306 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.771319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.772026 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.772026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.772952 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.772955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.773474 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.773475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.773724 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.774296 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:47.774305 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:47.775888 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:47.777870 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.777877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.777879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.778100 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.778347 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:47.778348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:47.778348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:47.778462 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:47.778729 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:47.778736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:47.778880 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.778994 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.779163 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":118,"threads_started":1}
I20260810 04:18:47.779948 31979 heartbeater.cc:347] Connected to a master server at 127.15.216.126:32953
I20260810 04:18:47.780061 31979 heartbeater.cc:464] Registering TS with master...
I20260810 04:18:47.780246 31979 heartbeater.cc:511] Master 127.15.216.126:32953 requested a full tablet report, sending...
I20260810 04:18:47.780695 31486 ts_manager.cc:194] Registered new tserver with Master: 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545)
I20260810 04:18:47.781169 31486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:60115
I20260810 04:18:47.781198 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.773617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.15.216.126:32953)
0810 04:18:47.773744 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.773746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.773835 (+ 89us) server_negotiation.cc:407] Connection header received
0810 04:18:47.773874 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.773875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.773889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.773964 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:47.774935 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.775456 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.776387 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.776651 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.776705 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.778017 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:47.778018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:47.778019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:47.778025 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:47.779931 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.780290 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.780291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.780291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.780350 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:47.780543 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:47.780544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:47.780544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:47.780684 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:47.780866 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.780940 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.781020 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260810 04:18:47.781991 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:18:47.784627 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.782330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.15.216.126:32953)
0810 04:18:47.782398 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.782400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.782477 (+ 77us) server_negotiation.cc:407] Connection header received
0810 04:18:47.782556 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.782557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.782593 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.782666 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:18:47.782756 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.783238 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.783967 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.784145 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.784193 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.784350 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:18:47.784431 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:18:47.784491 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.784524 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.784543 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:18:47.784868 31486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:18:47.786484 31486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:18:47.788647 31615 tablet_service.cc:1467] Processing CreateTablet for tablet 1b687636c73e40e180099688ec1adec5 (DEFAULT_TABLE table=test-table [id=686f30585eeb4810a383623939bbd317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:47.788867 31615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b687636c73e40e180099688ec1adec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:47.790563 31571 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap starting.
I20260810 04:18:47.790951 31571 tablet_bootstrap.cc:654] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:47.792047 31571 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: No bootstrap required, opened a new log
I20260810 04:18:47.792115 31571 ts_tablet_manager.cc:1397] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:18:47.792361 31571 raft_consensus.cc:348] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.792459 31571 raft_consensus.cc:374] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:47.792517 31571 raft_consensus.cc:729] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd7b7bc9d9c4f2a8dca9ae08117474f, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.792570 31571 consensus_queue.cc:260] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.792608 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.788338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:36765 (local address 127.0.0.1:36622)
0810 04:18:47.788586 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.788597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.788640 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.788872 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.788873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.788875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.788925 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.788927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.790275 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.790276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.791444 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.791448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.792328 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.792329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.792407 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.792428 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.792480 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.792528 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260810 04:18:47.792677 31571 ts_tablet_manager.cc:1428] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 04:18:47.792968 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.788445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.15.216.66:36765)
0810 04:18:47.788641 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.788646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.788661 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:18:47.788708 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.788711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.788730 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.788861 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.789022 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.790142 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.791546 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.792221 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.792326 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.792389 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.792808 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.792857 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
I20260810 04:18:47.793715 31769 tablet_service.cc:1467] Processing CreateTablet for tablet 1b687636c73e40e180099688ec1adec5 (DEFAULT_TABLE table=test-table [id=686f30585eeb4810a383623939bbd317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:47.794132 31769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b687636c73e40e180099688ec1adec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:47.794425 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.788357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:38545 (local address 127.0.0.1:43858)
0810 04:18:47.790585 (+ 2228us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.790597 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.790651 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.790875 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.790876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.790878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.790919 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.790921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.792007 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.792008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.793195 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.793198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.794169 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.794170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.794240 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.794244 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.794345 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.794377 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2208,"thread_start_us":119,"threads_started":1}
I20260810 04:18:47.794498 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.788672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.15.216.67:38545)
0810 04:18:47.788860 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.788864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.790709 (+ 1845us) server_negotiation.cc:407] Connection header received
0810 04:18:47.790758 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.790760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.790772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.790862 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.790982 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.791892 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.793310 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.794073 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.794165 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.794226 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.794365 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.794408 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260810 04:18:47.795250 31910 tablet_service.cc:1467] Processing CreateTablet for tablet 1b687636c73e40e180099688ec1adec5 (DEFAULT_TABLE table=test-table [id=686f30585eeb4810a383623939bbd317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:18:47.795550 31910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b687636c73e40e180099688ec1adec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:18:47.797304 31989 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Bootstrap starting.
I20260810 04:18:47.798007 31989 tablet_bootstrap.cc:654] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:47.798534 31989 log.cc:824] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:47.798547 31990 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Bootstrap starting.
I20260810 04:18:47.799122 31990 tablet_bootstrap.cc:654] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Neither blocks nor log segments found. Creating new log.
I20260810 04:18:47.799365 31989 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: No bootstrap required, opened a new log
I20260810 04:18:47.799454 31989 ts_tablet_manager.cc:1397] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:18:47.799647 31990 log.cc:824] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:47.800457 31990 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: No bootstrap required, opened a new log
I20260810 04:18:47.800534 31990 ts_tablet_manager.cc:1397] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:18:47.801296 31989 raft_consensus.cc:348] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.801458 31989 raft_consensus.cc:374] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:47.801509 31989 raft_consensus.cc:729] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff79c3242b04e33a80673f792c0332b, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.801676 31989 consensus_queue.cc:260] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.802026 31989 ts_tablet_manager.cc:1428] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 04:18:47.802066 31840 heartbeater.cc:503] Master 127.15.216.126:32953 was elected leader, sending a full tablet report...
I20260810 04:18:47.802093 31990 raft_consensus.cc:348] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.802242 31990 raft_consensus.cc:374] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:18:47.802276 31990 raft_consensus.cc:729] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4cc3000d434582b529d498123b5605, State: Initialized, Role: FOLLOWER
I20260810 04:18:47.802404 31990 consensus_queue.cc:260] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.802726 31990 ts_tablet_manager.cc:1428] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:18:47.802749 31979 heartbeater.cc:503] Master 127.15.216.126:32953 was elected leader, sending a full tablet report...
I20260810 04:18:47.845805 31994 raft_consensus.cc:482] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:47.845949 31994 raft_consensus.cc:504] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.846419 31994 leader_election.cc:288] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765), 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891)
I20260810 04:18:47.849488 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.846768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:46881 (local address 127.15.216.65:45891)
0810 04:18:47.846817 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.846819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.846862 (+ 43us) server_negotiation.cc:407] Connection header received
0810 04:18:47.846893 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.846894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.846923 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.846983 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.847151 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.847683 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.848717 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.849146 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.849237 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.849320 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.849377 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.849417 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:18:47.849486 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.846594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:45891 (local address 127.15.216.67:46881)
0810 04:18:47.846740 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.846754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.846814 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.846987 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.846988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.846989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.847065 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.847066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.847777 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.847778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.848605 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.848607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.849235 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.849236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.849312 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.849316 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.849372 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.849413 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":66}
I20260810 04:18:47.849666 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.846661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:44485 (local address 127.15.216.66:36765)
0810 04:18:47.846909 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.846912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.846924 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:18:47.846962 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.846963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.846971 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.847049 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:18:47.847246 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.847778 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.848843 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.849274 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.849367 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.849376 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.849535 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.849575 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260810 04:18:47.849704 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.846594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:36765 (local address 127.15.216.67:44485)
0810 04:18:47.846819 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.846830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:47.846903 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:47.847056 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:47.847057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:47.847059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:18:47.847152 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.847153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.847882 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.847883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.848737 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:47.848739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.849368 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.849369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:47.849449 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.849452 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:18:47.849528 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 04:18:47.849571 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":16,"thread_start_us":120,"threads_started":1}
I20260810 04:18:47.849838 31642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b687636c73e40e180099688ec1adec5" candidate_uuid: "3d4cc3000d434582b529d498123b5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" is_pre_election: true
I20260810 04:18:47.850055 31642 raft_consensus.cc:2454] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d4cc3000d434582b529d498123b5605 in term 0.
I20260810 04:18:47.850081 31790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b687636c73e40e180099688ec1adec5" candidate_uuid: "3d4cc3000d434582b529d498123b5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff79c3242b04e33a80673f792c0332b" is_pre_election: true
I20260810 04:18:47.850227 31790 raft_consensus.cc:2454] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d4cc3000d434582b529d498123b5605 in term 0.
I20260810 04:18:47.850355 31860 leader_election.cc:302] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d4cc3000d434582b529d498123b5605, 8fd7b7bc9d9c4f2a8dca9ae08117474f; no voters:
I20260810 04:18:47.850544 31994 raft_consensus.cc:2781] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:18:47.850622 31994 raft_consensus.cc:482] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:18:47.850660 31994 raft_consensus.cc:3037] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:47.851441 31994 raft_consensus.cc:504] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.851606 31994 leader_election.cc:288] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [CANDIDATE]: Term 1 election: Requested vote from peers 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765), 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891)
I20260810 04:18:47.851786 31790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b687636c73e40e180099688ec1adec5" candidate_uuid: "3d4cc3000d434582b529d498123b5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff79c3242b04e33a80673f792c0332b"
I20260810 04:18:47.851886 31790 raft_consensus.cc:3037] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:47.851838 31642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b687636c73e40e180099688ec1adec5" candidate_uuid: "3d4cc3000d434582b529d498123b5605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f"
I20260810 04:18:47.851971 31642 raft_consensus.cc:3037] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:18:47.852687 31790 raft_consensus.cc:2454] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d4cc3000d434582b529d498123b5605 in term 1.
I20260810 04:18:47.852874 31642 raft_consensus.cc:2454] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d4cc3000d434582b529d498123b5605 in term 1.
I20260810 04:18:47.853075 31861 leader_election.cc:302] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b; no voters:
I20260810 04:18:47.853178 31994 raft_consensus.cc:2781] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:18:47.853386 31994 raft_consensus.cc:686] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 1 LEADER]: Becoming Leader. State: Replica: 3d4cc3000d434582b529d498123b5605, State: Running, Role: LEADER
I20260810 04:18:47.853487 31994 consensus_queue.cc:237] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:47.854391 31486 catalog_manager.cc:5495] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d4cc3000d434582b529d498123b5605 (127.15.216.67). New cstate: current_term: 1 leader_uuid: "3d4cc3000d434582b529d498123b5605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } health_report { overall_health: UNKNOWN } } }
I20260810 04:18:47.867070 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.864577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.15.216.67:38545)
0810 04:18:47.864648 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:47.864650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:47.864660 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:18:47.864701 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:47.864701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:47.864721 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:47.864827 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:18:47.864937 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.865446 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.866243 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:47.866461 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:47.866539 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:47.866681 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:18:47.866796 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:18:47.866893 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:47.866912 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 04:18:47.866934 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 04:18:47.869294 31642 raft_consensus.cc:1261] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Refusing update from remote peer 3d4cc3000d434582b529d498123b5605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:47.869321 31790 raft_consensus.cc:1261] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 1 FOLLOWER]: Refusing update from remote peer 3d4cc3000d434582b529d498123b5605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:18:47.869752 31994 consensus_queue.cc:1035] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:47.869964 31996 consensus_queue.cc:1035] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:18:47.872586 32001 mvcc.cc:205] Tried to move back new op lower bound from 7316830322148540416 to 7316830322088775680. Current Snapshot: MvccSnapshot[applied={T|T < 7316830322148540416}]
I20260810 04:18:47.872609 32002 mvcc.cc:205] Tried to move back new op lower bound from 7316830322148540416 to 7316830322088775680. Current Snapshot: MvccSnapshot[applied={T|T < 7316830322148540416}]
I20260810 04:18:47.872969 32000 mvcc.cc:205] Tried to move back new op lower bound from 7316830322148540416 to 7316830322088775680. Current Snapshot: MvccSnapshot[applied={T|T < 7316830322148540416}]
I20260810 04:18:47.875108 31486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:18:47.875207 31486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:18:47.875939 31486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Sending DeleteTablet for 3 replicas of tablet 1b687636c73e40e180099688ec1adec5
I20260810 04:18:47.876269 31615 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b687636c73e40e180099688ec1adec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:47 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:54388
I20260810 04:18:47.876255 31769 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b687636c73e40e180099688ec1adec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:47 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:36622
I20260810 04:18:47.876340 31910 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b687636c73e40e180099688ec1adec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:18:47 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43858
I20260810 04:18:47.876494 32011 tablet_replica.cc:330] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: stopping tablet replica
I20260810 04:18:47.876529 31698 tablet_replica.cc:330] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: stopping tablet replica
I20260810 04:18:47.876587 31698 raft_consensus.cc:2229] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:18:47.876577 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 31526
I20260810 04:18:47.876590 32011 raft_consensus.cc:2229] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:18:47.876645 31698 raft_consensus.cc:2258] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:18:47.876689 32011 raft_consensus.cc:2258] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:18:47.878978 32011 ts_tablet_manager.cc:1905] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:18:47.880774 32011 ts_tablet_manager.cc:1918] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:18:47.880908 32011 log.cc:1196] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/1b687636c73e40e180099688ec1adec5
I20260810 04:18:47.881412 32011 ts_tablet_manager.cc:1939] T 1b687636c73e40e180099688ec1adec5 P 7ff79c3242b04e33a80673f792c0332b: Deleting consensus metadata
I20260810 04:18:47.881903 31471 catalog_manager.cc:4841] TS 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765): tablet 1b687636c73e40e180099688ec1adec5 (table test-table [id=686f30585eeb4810a383623939bbd317]) successfully deleted
I20260810 04:18:47.885749 32012 tablet_replica.cc:330] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: stopping tablet replica
I20260810 04:18:47.885901 32012 raft_consensus.cc:2229] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:18:47.886078 32012 raft_consensus.cc:2258] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 04:18:47.886528 31470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:45891 (error 108)
I20260810 04:18:47.886605 32012 ts_tablet_manager.cc:1905] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:18:47.886749 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 31691
I20260810 04:18:47.887284 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:47.887143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:45891 (local address 127.0.0.1:54394)
0810 04:18:47.887198 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:47.887247 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:45891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 04:18:47.887473 31470 catalog_manager.cc:4584] TS 8fd7b7bc9d9c4f2a8dca9ae08117474f (127.15.216.65:45891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b687636c73e40e180099688ec1adec5: Network error: Client connection negotiation failed: client connection to 127.15.216.65:45891: connect: Connection refused (error 111)
I20260810 04:18:47.888700 32012 ts_tablet_manager.cc:1918] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:18:47.888784 32012 log.cc:1196] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/1b687636c73e40e180099688ec1adec5
I20260810 04:18:47.889098 32012 ts_tablet_manager.cc:1939] T 1b687636c73e40e180099688ec1adec5 P 3d4cc3000d434582b529d498123b5605: Deleting consensus metadata
I20260810 04:18:47.889634 31472 catalog_manager.cc:4841] TS 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545): tablet 1b687636c73e40e180099688ec1adec5 (table test-table [id=686f30585eeb4810a383623939bbd317]) successfully deleted
I20260810 04:18:47.895542 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 31843
I20260810 04:18:47.902468 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 31448
I20260810 04:18:47.909158 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:32953
--webserver_interface=127.15.216.126
--webserver_port=40213
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:18:48.000301 32014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:48.000507 32014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:48.000530 32014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:48.001708 32014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:18:48.001749 32014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:48.001763 32014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:48.001776 32014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:18:48.001832 32014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:18:48.003310 32014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:32953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:48.003569 32014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:48.003808 32014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:18:48.009194 32014 server_base.cc:1034] running on GCE node
W20260810 04:18:48.009245 32022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.009160 32024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.009153 32020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:48.009770 32014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:48.010074 32014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:48.011243 32014 hybrid_clock.cc:648] HybridClock initialized: now 1786335528011231 us; error 47 us; skew 500 ppm
Aug 10 04:18:48 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:48.014086 32014 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:18:48.014598 32014 webserver.cc:460] Webserver started at http://127.15.216.126:40213/ using document root <none> and password file <none>
I20260810 04:18:48.014861 32014 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:48.014933 32014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:48.016396 32014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:48.017132 32032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.017413 32014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:48.017513 32014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "61972f53d4b34f2d9d834eb231892574"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "bb06b5e29f340246f54f9de303d0e1ed"
server_key_iv: "62d98d8430c7896610bd27f7df42bc06"
server_key_version: "encryptionkey@0"
I20260810 04:18:48.017925 32014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:48.031162 32014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:48.033495 32014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:48.033704 32014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:48.037719 32014 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:32953
I20260810 04:18:48.037763 32086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:32953 every 8 connection(s)
I20260810 04:18:48.038156 32014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:18:48.039881 32087 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:18:48.040548 32087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Bootstrap starting.
I20260810 04:18:48.045226 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 32014
I20260810 04:18:48.045612 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:45891
--local_ip_for_outbound_sockets=127.15.216.65
--tserver_master_addrs=127.15.216.126:32953
--webserver_port=39795
--webserver_interface=127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:48.050843 32087 log.cc:824] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:48.053133 32087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:18:48.053462 32087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574: Bootstrap complete.
I20260810 04:18:48.055883 32087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:48.056154 32087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61972f53d4b34f2d9d834eb231892574, State: Initialized, Role: FOLLOWER
I20260810 04:18:48.056363 32087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:48.056464 32087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:18:48.056520 32087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:18:48.056569 32087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:18:48.057560 32087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:48.057708 32087 leader_election.cc:302] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61972f53d4b34f2d9d834eb231892574; no voters:
I20260810 04:18:48.057921 32087 leader_election.cc:288] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 04:18:48.057961 32091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 04:18:48.058079 32091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [term 3 LEADER]: Becoming Leader. State: Replica: 61972f53d4b34f2d9d834eb231892574, State: Running, Role: LEADER
I20260810 04:18:48.058177 32091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } }
I20260810 04:18:48.058205 32087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:18:48.058466 32092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "61972f53d4b34f2d9d834eb231892574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } } }
I20260810 04:18:48.058559 32092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:48.058774 32095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:18:48.059705 32095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:18:48.059820 32093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61972f53d4b34f2d9d834eb231892574. Latest consensus state: current_term: 3 leader_uuid: "61972f53d4b34f2d9d834eb231892574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61972f53d4b34f2d9d834eb231892574" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 32953 } } }
I20260810 04:18:48.059890 32093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61972f53d4b34f2d9d834eb231892574 [sys.catalog]: This master's current role is: LEADER
I20260810 04:18:48.059993 32095 catalog_manager.cc:1259] Loaded cluster ID: 39ddaeab0a734db8947177d580543c7e
I20260810 04:18:48.060163 32095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 04:18:48.060751 32095 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb34f4e39ad google::LogMessage::Fail() at ??:0
@ 0x7fb34f4e688a google::LogMessage::SendToLog() at ??:0
@ 0x7fb34f4e34de google::LogMessage::Flush() at ??:0
@ 0x7fb34f4e4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb3574080f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb3574118b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb3513b10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb3513a6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb35220a6db start_thread at ??:0
@ 0x7fb34dfe161f clone at ??:0
W20260810 04:18:48.147979 32089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:48.148301 32089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:48.148353 32089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:48.150503 32089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:48.150568 32089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:48.150612 32089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:18:48.153255 32089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:45891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=39795
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:48.153545 32089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:48.153836 32089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:48.159791 32124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.159822 32125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.159822 32127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:48.160144 32089 server_base.cc:1034] running on GCE node
I20260810 04:18:48.160377 32089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:48.160630 32089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:48.161780 32089 hybrid_clock.cc:648] HybridClock initialized: now 1786335528161759 us; error 36 us; skew 500 ppm
Aug 10 04:18:48 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:48.164711 32089 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:18:48.165256 32089 webserver.cc:460] Webserver started at http://127.15.216.65:39795/ using document root <none> and password file <none>
I20260810 04:18:48.165534 32089 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:48.165583 32089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:48.167250 32089 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:18:48.168033 32134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.168267 32089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:18:48.168412 32089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "9b2196167e6a9ac144b120cad31fb128"
server_key_iv: "0b4a699f35c9769c5836f65a71eaf886"
server_key_version: "encryptionkey@0"
I20260810 04:18:48.168763 32089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:48.193115 32089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:48.195652 32089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:48.195842 32089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:48.196141 32089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:48.196643 32141 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:18:48.198745 32089 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:18:48.198824 32089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:18:48.198871 32089 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:18:48.199630 32089 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:18:48.199692 32089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:18:48.199765 32141 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap starting.
I20260810 04:18:48.207382 32089 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:45891
I20260810 04:18:48.207775 32089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:18:48.208364 32257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:45891 every 8 connection(s)
I20260810 04:18:48.212394 32141 log.cc:824] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:18:48.212800 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 32089
I20260810 04:18:48.213109 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:36765
--local_ip_for_outbound_sockets=127.15.216.66
--tserver_master_addrs=127.15.216.126:32953
--webserver_port=36767
--webserver_interface=127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:48.215382 32141 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:18:48.215713 32141 tablet_bootstrap.cc:492] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Bootstrap complete.
I20260810 04:18:48.217151 32141 ts_tablet_manager.cc:1397] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20260810 04:18:48.218808 32141 raft_consensus.cc:348] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:48.219161 32141 raft_consensus.cc:729] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd7b7bc9d9c4f2a8dca9ae08117474f, State: Initialized, Role: FOLLOWER
I20260810 04:18:48.219343 32141 consensus_queue.cc:260] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:48.219683 32141 ts_tablet_manager.cc:1428] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260810 04:18:48.263101 32260 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:18:48.208546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:42693)
0810 04:18:48.208738 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.208776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:48.209254 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:48.262886 (+ 53632us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
W20260810 04:18:48.263401 32258 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:32953 (0 consecutive failures): Network error: Failed to ping master at 127.15.216.126:32953: Client connection negotiation failed: client connection to 127.15.216.126:32953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:18:48.263747 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.263625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:35747)
0810 04:18:48.263681 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.263723 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:18:48.264119 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.264027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:54891)
0810 04:18:48.264073 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.264100 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:18:48.264245 32258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:18:48.302889 32262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:48.303097 32262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:48.303120 32262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:48.304260 32262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:48.304301 32262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:48.304327 32262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:18:48.305848 32262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:36765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=36767
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:48.306084 32262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:48.306325 32262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:48.311580 32272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.311626 32269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.311597 32270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:48.311785 32262 server_base.cc:1034] running on GCE node
I20260810 04:18:48.312004 32262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:48.312252 32262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:48.313416 32262 hybrid_clock.cc:648] HybridClock initialized: now 1786335528313404 us; error 26 us; skew 500 ppm
Aug 10 04:18:48 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:48.316195 32262 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:18:48.316676 32262 webserver.cc:460] Webserver started at http://127.15.216.66:36767/ using document root <none> and password file <none>
I20260810 04:18:48.316908 32262 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:48.316956 32262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:48.318488 32262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:18:48.319207 32279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.319451 32262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:48.319531 32262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "7ff79c3242b04e33a80673f792c0332b"
format_stamp: "Formatted at 2026-08-10 04:18:46 on dist-test-slave-g350"
server_key: "ddb7f1fbb572d95ce162b573da7377ba"
server_key_iv: "96bc8bdd84fc6cb94e11e7cba270156d"
server_key_version: "encryptionkey@0"
I20260810 04:18:48.319865 32262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:48.340809 32262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:48.343027 32262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:48.343147 32262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:48.343328 32262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:48.343649 32262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:48.343684 32262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.343709 32262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:48.343755 32262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.350172 32262 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:36765
I20260810 04:18:48.350217 32401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:36765 every 8 connection(s)
I20260810 04:18:48.350632 32262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:18:48.351279 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 32262
I20260810 04:18:48.351552 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:38545
--local_ip_for_outbound_sockets=127.15.216.67
--tserver_master_addrs=127.15.216.126:32953
--webserver_port=42001
--webserver_interface=127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:18:48.351919 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.351411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:36485)
0810 04:18:48.351622 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.351741 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
W20260810 04:18:48.352337 32402 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:32953 (0 consecutive failures): Network error: Failed to ping master at 127.15.216.126:32953: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
I20260810 04:18:48.352665 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.352562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:37405)
0810 04:18:48.352617 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.352645 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 04:18:48.352932 32404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.352876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:50901)
0810 04:18:48.352900 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.352919 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260810 04:18:48.353052 32402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:18:48.435981 32405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:18:48.436185 32405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:18:48.436218 32405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:18:48.437285 32405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:18:48.437317 32405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:18:48.437342 32405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:18:48.438738 32405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:38545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=42001
--tserver_master_addrs=127.15.216.126:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:18:48.438925 32405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:18:48.439097 32405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:18:48.444327 32411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.444442 32410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:18:48.444337 32413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:18:48.444537 32405 server_base.cc:1034] running on GCE node
I20260810 04:18:48.444818 32405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:18:48.445019 32405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:18:48.446184 32405 hybrid_clock.cc:648] HybridClock initialized: now 1786335528446168 us; error 30 us; skew 500 ppm
Aug 10 04:18:48 dist-test-slave-g350 krb5kdc[30858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:18:48.449080 32405 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:18:48.449628 32405 webserver.cc:460] Webserver started at http://127.15.216.67:42001/ using document root <none> and password file <none>
I20260810 04:18:48.449878 32405 fs_manager.cc:359] Metadata directory not provided
I20260810 04:18:48.449951 32405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:18:48.451396 32405 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:48.452107 32421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.452353 32405 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:18:48.452528 32405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "3d4cc3000d434582b529d498123b5605"
format_stamp: "Formatted at 2026-08-10 04:18:47 on dist-test-slave-g350"
server_key: "6d3b3f20719fcdf58622de7e551da229"
server_key_iv: "07df2c4a9c21db3905dd41534e0cba9f"
server_key_version: "encryptionkey@0"
I20260810 04:18:48.452893 32405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:18:48.466672 32405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:18:48.469247 32405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:18:48.469398 32405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:18:48.469743 32405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:18:48.470077 32405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:18:48.470135 32405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.470178 32405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:18:48.470206 32405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:18:48.476542 32405 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:38545
I20260810 04:18:48.476615 32545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:38545 every 8 connection(s)
I20260810 04:18:48.476943 32405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:18:48.477815 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 32405
I20260810 04:18:48.478155 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.477665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:46035)
0810 04:18:48.477839 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.477980 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260810 04:18:48.478667 32546 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:32953 (0 consecutive failures): Network error: Failed to ping master at 127.15.216.126:32953: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
I20260810 04:18:48.478986 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.478888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:43433)
0810 04:18:48.478935 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.478964 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 04:18:48.479301 32548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:48.479211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:36047)
0810 04:18:48.479258 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:48.479282 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:18:48.479434 32546 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:18:49.264869 32550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:49.264613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:43543)
0810 04:18:49.264759 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:49.264802 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260810 04:18:49.353798 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:49.353526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:40787)
0810 04:18:49.353685 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:49.353725 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260810 04:18:49.480015 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:49.479790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:53331)
0810 04:18:49.479923 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:49.479961 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260810 04:18:49.766796 32554 raft_consensus.cc:482] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:49.766961 32554 raft_consensus.cc:504] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:49.767383 32554 leader_election.cc:288] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765), 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545)
Aug 10 04:18:49 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.67@KRBTEST.COM
Aug 10 04:18:49 dist-test-slave-g350 krb5kdc[30858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.66@KRBTEST.COM
I20260810 04:18:49.778204 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:49.767821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:57297 (local address 127.15.216.67:38545)
0810 04:18:49.767974 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:49.767986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:49.768020 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:18:49.768468 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:49.768470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:49.768507 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:49.768662 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:49.769339 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.770254 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.771315 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.771717 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.771889 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:49.774413 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:49.774418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:49.774418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:49.774433 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:49.776331 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:49.776927 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:49.776930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:49.776930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:49.777017 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:49.777287 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:49.777288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:49.777289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:49.777531 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:49.777668 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:49.777955 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 04:18:49.778100 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260810 04:18:49.778190 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:49.767573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:38545 (local address 127.15.216.65:57297)
0810 04:18:49.767715 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:49.767725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:49.767984 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:49.768683 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:49.768685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:49.769036 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:49.769218 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:49.769224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.770360 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.770361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:49.771206 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:49.771209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.771812 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.771813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:49.771920 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:49.772511 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:49.772522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:49.774289 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:49.776446 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:49.776454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:49.776456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:49.776793 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:49.777092 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:49.777093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:49.777093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:49.777209 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:49.777631 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:49.777638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:49.777775 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 04:18:49.777881 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:18:49.778078 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":8,"thread_start_us":82,"threads_started":1}
I20260810 04:18:49.778256 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:49.767676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:50775 (local address 127.15.216.66:36765)
0810 04:18:49.767850 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 04:18:49.767862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 04:18:49.767903 (+ 41us) server_negotiation.cc:407] Connection header received
0810 04:18:49.768444 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:18:49.768446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:18:49.768491 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:18:49.768638 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:18:49.769339 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.770265 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.771319 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.771876 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.772026 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:49.774800 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:18:49.774804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:18:49.774804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:18:49.774814 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:18:49.776433 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:49.777026 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:49.777030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:49.777030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:49.777127 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:18:49.777351 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:18:49.777352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:18:49.777352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:18:49.777634 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:18:49.777735 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 04:18:49.777962 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 04:18:49.778125 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":55,"threads_started":1}
I20260810 04:18:49.778303 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:49.767563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:36765 (local address 127.15.216.65:50775)
0810 04:18:49.767709 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:49.767721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:18:49.767795 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:18:49.768681 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:18:49.768684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:18:49.769036 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:18:49.769218 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:49.769224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.770386 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.770387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:49.771208 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:18:49.771211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:18:49.771988 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:18:49.771990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:18:49.772058 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:18:49.772662 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:18:49.772667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:18:49.774680 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:18:49.776668 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:49.776669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:49.776670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:49.776939 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:49.777205 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:18:49.777206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:18:49.777207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:18:49.777269 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:18:49.777722 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:18:49.777723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:18:49.777794 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 04:18:49.777890 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 04:18:49.778166 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":204,"thread_start_us":56,"threads_started":1}
W20260810 04:18:49.779131 32137 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
W20260810 04:18:49.779611 32138 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
I20260810 04:18:49.779675 32138 leader_election.cc:302] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fd7b7bc9d9c4f2a8dca9ae08117474f; no voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b
I20260810 04:18:49.779788 32554 raft_consensus.cc:2726] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:18:50.265722 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:50.265566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:34303)
0810 04:18:50.265640 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:50.265680 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 04:18:50.354539 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:50.354282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:53893)
0810 04:18:50.354446 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:50.354493 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260810 04:18:50.480711 32560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:50.480509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:40395)
0810 04:18:50.480628 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:50.480668 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260810 04:18:51.266546 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:51.266244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:35923)
0810 04:18:51.266453 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:51.266495 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
I20260810 04:18:51.355247 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:51.355039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:51603)
0810 04:18:51.355166 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:51.355201 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260810 04:18:51.481503 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:51.481184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:48401)
0810 04:18:51.481404 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:51.481448 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":131,"threads_started":1}
I20260810 04:18:51.908548 32564 raft_consensus.cc:482] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:51.908651 32564 raft_consensus.cc:504] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:51.908880 32564 leader_election.cc:288] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765), 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545)
W20260810 04:18:51.909447 32137 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
W20260810 04:18:51.909519 32138 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
I20260810 04:18:51.909549 32137 leader_election.cc:302] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fd7b7bc9d9c4f2a8dca9ae08117474f; no voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b
I20260810 04:18:51.909643 32564 raft_consensus.cc:2726] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:18:52.267362 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:52.267115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:51851)
0810 04:18:52.267243 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:52.267295 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260810 04:18:52.356006 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:52.355751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:33675)
0810 04:18:52.355888 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:52.355939 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260810 04:18:52.482268 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:52.482006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:32843)
0810 04:18:52.482166 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:52.482220 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260810 04:18:53.268095 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:53.267831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:35443)
0810 04:18:53.268000 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:53.268039 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260810 04:18:53.356798 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:53.356544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:57091)
0810 04:18:53.356691 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:53.356750 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260810 04:18:53.483058 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:53.482801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:57633)
0810 04:18:53.482959 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:53.483008 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260810 04:18:53.727872 32572 raft_consensus.cc:482] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:53.727974 32572 raft_consensus.cc:504] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:53.728171 32572 leader_election.cc:288] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765), 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545)
W20260810 04:18:53.728654 32137 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
W20260810 04:18:53.728749 32138 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
I20260810 04:18:53.728816 32138 leader_election.cc:302] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fd7b7bc9d9c4f2a8dca9ae08117474f; no voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b
I20260810 04:18:53.728919 32572 raft_consensus.cc:2726] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:18:54.268896 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:54.268655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:51817)
0810 04:18:54.268804 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:54.268847 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260810 04:18:54.357664 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:54.357348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:41941)
0810 04:18:54.357528 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:54.357586 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260810 04:18:54.483914 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:54.483638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:45135)
0810 04:18:54.483814 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:54.483866 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":113,"threads_started":1}
I20260810 04:18:55.269802 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:55.269506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:57107)
0810 04:18:55.269704 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:55.269749 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260810 04:18:55.358443 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:55.358183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:40461)
0810 04:18:55.358347 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:55.358391 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260810 04:18:55.484673 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:55.484422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:46863)
0810 04:18:55.484587 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:55.484623 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260810 04:18:56.270555 32580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:56.270311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:53865)
0810 04:18:56.270463 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:56.270505 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260810 04:18:56.359205 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:56.358960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:35333)
0810 04:18:56.359103 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:56.359148 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260810 04:18:56.485554 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:56.485294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:34079)
0810 04:18:56.485456 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:56.485503 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260810 04:18:56.973224 32583 raft_consensus.cc:482] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:18:56.973354 32583 raft_consensus.cc:504] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:18:56.973568 32583 leader_election.cc:288] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765), 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545)
W20260810 04:18:56.974026 32138 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
W20260810 04:18:56.974110 32137 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
I20260810 04:18:56.974155 32137 leader_election.cc:302] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fd7b7bc9d9c4f2a8dca9ae08117474f; no voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b
I20260810 04:18:56.974251 32583 raft_consensus.cc:2726] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:18:57.271258 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:57.271012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:45059)
0810 04:18:57.271161 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:57.271205 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260810 04:18:57.360049 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:57.359781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:48251)
0810 04:18:57.359956 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:57.359999 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260810 04:18:57.486435 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:57.486127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:46745)
0810 04:18:57.486339 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:57.486386 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260810 04:18:58.272019 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:58.271802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:58683)
0810 04:18:58.271939 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:58.271972 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260810 04:18:58.360735 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:58.360511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:40813)
0810 04:18:58.360645 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:58.360686 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260810 04:18:58.487169 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:58.486921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:38513)
0810 04:18:58.487077 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:58.487121 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260810 04:18:59.272862 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:59.272553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:42557)
0810 04:18:59.272765 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:59.272812 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":65,"threads_started":1}
I20260810 04:18:59.361421 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:59.361191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:33523)
0810 04:18:59.361335 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:59.361372 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260810 04:18:59.487910 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:18:59.487662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:42985)
0810 04:18:59.487830 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:18:59.487862 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260810 04:19:00.273707 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:00.273425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:55783)
0810 04:19:00.273563 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:00.273652 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260810 04:19:00.362190 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:00.361903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:40261)
0810 04:19:00.362065 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:00.362135 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260810 04:19:00.488880 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:00.488595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:48025)
0810 04:19:00.488777 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:00.488829 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260810 04:19:01.274566 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:01.274259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:41581)
0810 04:19:01.274442 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:01.274490 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260810 04:19:01.362893 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:01.362654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:50971)
0810 04:19:01.362803 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:01.362843 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260810 04:19:01.489914 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:01.489679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:52063)
0810 04:19:01.489830 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:01.489865 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260810 04:19:02.275419 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:02.275190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:33669)
0810 04:19:02.275332 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:02.275368 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260810 04:19:02.363670 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:02.363449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:35917)
0810 04:19:02.363575 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:02.363615 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260810 04:19:02.490733 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:02.490486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:48637)
0810 04:19:02.490623 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:02.490664 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260810 04:19:03.276294 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:03.276059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.65:33875)
0810 04:19:03.276202 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:03.276244 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260810 04:19:03.350188 32604 raft_consensus.cc:482] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:03.350378 32604 raft_consensus.cc:504] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff79c3242b04e33a80673f792c0332b" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 36765 } } peers { permanent_uuid: "3d4cc3000d434582b529d498123b5605" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 38545 } } peers { permanent_uuid: "8fd7b7bc9d9c4f2a8dca9ae08117474f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 45891 } }
I20260810 04:19:03.350615 32604 leader_election.cc:288] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765), 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545)
W20260810 04:19:03.351091 32137 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7ff79c3242b04e33a80673f792c0332b (127.15.216.66:36765): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
W20260810 04:19:03.351174 32138 leader_election.cc:341] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4cc3000d434582b529d498123b5605 (127.15.216.67:38545): Not found: Tablet not found: 1b687636c73e40e180099688ec1adec5
I20260810 04:19:03.351255 32138 leader_election.cc:302] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8fd7b7bc9d9c4f2a8dca9ae08117474f; no voters: 3d4cc3000d434582b529d498123b5605, 7ff79c3242b04e33a80673f792c0332b
I20260810 04:19:03.351341 32604 raft_consensus.cc:2726] T 1b687636c73e40e180099688ec1adec5 P 8fd7b7bc9d9c4f2a8dca9ae08117474f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:19:03.364455 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:03.364244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.66:39731)
0810 04:19:03.364371 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:03.364405 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260810 04:19:03.480400 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 32089
I20260810 04:19:03.488858 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 32262
I20260810 04:19:03.491588 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:03.491278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:32953 (local address 127.15.216.67:51827)
0810 04:19:03.491494 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:03.491527 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:32953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":132,"threads_started":1}
I20260810 04:19:03.495811 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 32405
2026-08-10T04:19:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18351 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:19:03 dist-test-slave-g350 krb5kdc[32609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:19:03 dist-test-slave-g350 krb5kdc[32609](info): set up 2 sockets
Aug 10 04:19:03 dist-test-slave-g350 krb5kdc[32609](info): commencing operation
krb5kdc: starting...
W20260810 04:19:04.634691 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:19:04 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:19:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:19:04.765836 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41741
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:04.850383 32623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:04.850641 32623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:04.850685 32623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:04.851859 32623 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:19:04.851927 32623 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:19:04.851949 32623 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:19:04.851979 32623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:19:04.852026 32623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:04.852058 32623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:04.852087 32623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:19:04.852115 32623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:19:04.853715 32623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:04.853997 32623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:04.854252 32623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:04.859608 32628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:04.859608 32633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:04.859802 32623 server_base.cc:1034] running on GCE node
W20260810 04:19:04.859625 32630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:04.860225 32623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:04.860532 32623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:04.861670 32623 hybrid_clock.cc:648] HybridClock initialized: now 1786335544861635 us; error 38 us; skew 500 ppm
Aug 10 04:19:04 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335544, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:04.877759 32623 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:19:04.878381 32623 webserver.cc:460] Webserver started at http://127.15.216.126:38899/ using document root <none> and password file <none>
I20260810 04:19:04.878669 32623 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:04.878794 32623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:04.878971 32623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:19:04.880198 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "e5ef2b69afaa4404b4214f25d8b06087"
format_stamp: "Formatted at 2026-08-10 04:19:04 on dist-test-slave-g350"
server_key: "800d42379678694ba01af291e7c5a1ef"
server_key_iv: "fbdb6846ada38ba5fe0738c21dd5dc59"
server_key_version: "encryptionkey@0"
I20260810 04:19:04.880832 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "e5ef2b69afaa4404b4214f25d8b06087"
format_stamp: "Formatted at 2026-08-10 04:19:04 on dist-test-slave-g350"
server_key: "800d42379678694ba01af291e7c5a1ef"
server_key_iv: "fbdb6846ada38ba5fe0738c21dd5dc59"
server_key_version: "encryptionkey@0"
I20260810 04:19:04.882464 32623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260810 04:19:04.883442 32641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:04.883627 32623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:04.883746 32623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e5ef2b69afaa4404b4214f25d8b06087"
format_stamp: "Formatted at 2026-08-10 04:19:04 on dist-test-slave-g350"
server_key: "800d42379678694ba01af291e7c5a1ef"
server_key_iv: "fbdb6846ada38ba5fe0738c21dd5dc59"
server_key_version: "encryptionkey@0"
I20260810 04:19:04.883831 32623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:04.893271 32623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:04.895220 32623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:04.895383 32623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:04.899907 32623 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41741
I20260810 04:19:04.899967 32696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41741 every 8 connection(s)
I20260810 04:19:04.900336 32623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:19:04.901252 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:19:04.901516 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 32623
I20260810 04:19:04.901652 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:19:04.901888 16225 external_mini_cluster.cc:1428] Setting key aa27681dbc5243618a30d8bbcdef8bc5
I20260810 04:19:04.903967 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Bootstrap starting.
I20260810 04:19:04.904563 32697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Neither blocks nor log segments found. Creating new log.
I20260810 04:19:04.904886 32697 log.cc:824] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:04.905750 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: No bootstrap required, opened a new log
I20260810 04:19:04.908336 32697 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:04.908542 32697 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:19:04.908581 32697 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ef2b69afaa4404b4214f25d8b06087, State: Initialized, Role: FOLLOWER
I20260810 04:19:04.908715 32697 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:04.908790 32697 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:19:04.908838 32697 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:19:04.908890 32697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 04:19:04 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:19:04.909525 32697 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:04.909683 32697 leader_election.cc:302] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ef2b69afaa4404b4214f25d8b06087; no voters:
I20260810 04:19:04.909871 32697 leader_election.cc:288] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:19:04.909950 32702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:19:04.910137 32702 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 LEADER]: Becoming Leader. State: Replica: e5ef2b69afaa4404b4214f25d8b06087, State: Running, Role: LEADER
I20260810 04:19:04.910198 32697 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:19:04.910302 32702 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:04.910706 32702 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ef2b69afaa4404b4214f25d8b06087. Latest consensus state: current_term: 1 leader_uuid: "e5ef2b69afaa4404b4214f25d8b06087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } } }
I20260810 04:19:04.910977 32702 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: This master's current role is: LEADER
I20260810 04:19:04.910895 32704 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5ef2b69afaa4404b4214f25d8b06087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } } }
I20260810 04:19:04.911381 32704 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: This master's current role is: LEADER
I20260810 04:19:04.911387 32712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:19:04.912045 32712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:19:04.913743 32712 catalog_manager.cc:1347] Generated new cluster ID: dec24dcbad3242699f8e9cab514a1d7c
I20260810 04:19:04.913844 32712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:19:04.914000 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:04.902448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.15.216.126:41741)
0810 04:19:04.902653 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:04.902660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:04.902685 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:19:04.903096 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:04.903102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:04.903145 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:04.903298 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:04.903974 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:04.905124 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:04.906259 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:04.906687 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:04.906775 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:04.909742 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:04.909747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:04.909747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:04.909760 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:04.912160 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:04.912694 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:04.912697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:04.912698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:04.912801 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:04.913065 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:04.913067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:04.913067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:04.913330 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:04.913508 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:04.913631 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 04:19:04.913778 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260810 04:19:04.925505 32712 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:19:04.926231 32712 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:19:04.936282 32712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Generated new TSK 0
I20260810 04:19:04.936481 32712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:19:04.966948 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41741
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:05.051020 32724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:05.051251 32724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:05.051299 32724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:05.052446 32724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:05.052511 32724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:05.052557 32724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:19:05.054122 32724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:05.054396 32724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:05.054632 32724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:05.059895 32729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:05.059891 32733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:05.059911 32730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:05.060353 32724 server_base.cc:1034] running on GCE node
I20260810 04:19:05.060590 32724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:05.060845 32724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:05.062021 32724 hybrid_clock.cc:648] HybridClock initialized: now 1786335545062005 us; error 26 us; skew 500 ppm
Aug 10 04:19:05 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:05.064968 32724 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:19:05.065493 32724 webserver.cc:460] Webserver started at http://127.15.216.65:38161/ using document root <none> and password file <none>
I20260810 04:19:05.065809 32724 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:05.065884 32724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:05.066047 32724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:19:05.067222 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "e3dc229a78bd4717af6c964f9fa94300"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "8cb5e653b4f24dd3e873698994f5b82c"
server_key_iv: "0ee9e437ab93174de9c4d6affa745cce"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.067620 32724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "e3dc229a78bd4717af6c964f9fa94300"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "8cb5e653b4f24dd3e873698994f5b82c"
server_key_iv: "0ee9e437ab93174de9c4d6affa745cce"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.069061 32724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:19:05.069994 32740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.070178 32724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:19:05.070346 32724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "e3dc229a78bd4717af6c964f9fa94300"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "8cb5e653b4f24dd3e873698994f5b82c"
server_key_iv: "0ee9e437ab93174de9c4d6affa745cce"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.070437 32724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:05.090649 32724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:05.092792 32724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:05.092988 32724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:05.093255 32724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:05.093678 32724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:19:05.093745 32724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.093786 32724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:19:05.093811 32724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.100649 32724 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:38293
I20260810 04:19:05.100708 397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:38293 every 8 connection(s)
I20260810 04:19:05.101070 32724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:19:05.103211 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 32724
I20260810 04:19:05.103308 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:19:05.103529 16225 external_mini_cluster.cc:1428] Setting key a69fcc799ed867f9c25943a3bedf9206
Aug 10 04:19:05 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:19:05.112717 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.101909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:33011 (local address 127.15.216.126:41741)
0810 04:19:05.101980 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.101983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.102235 (+ 252us) server_negotiation.cc:407] Connection header received
0810 04:19:05.102731 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.102732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.102743 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.102805 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:05.104060 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.104649 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.106464 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.106799 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.106888 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.109396 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:05.109398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:05.109398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:05.109407 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:05.111246 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.111713 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.111715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.111715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.111781 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.112049 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.112050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.112050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.112171 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:05.112249 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.112525 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.112624 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:19:05.112911 400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.101881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:33011)
0810 04:19:05.102092 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.102132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.102638 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.102864 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.102868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.103648 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:05.103961 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.103968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.104772 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.104774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.106366 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.106371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.106903 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.106904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.107021 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.107622 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:05.107633 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:05.109293 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:05.111389 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:05.111397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:05.111400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:05.111630 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:05.111858 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:05.111859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:05.111859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:05.111970 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:05.112240 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:05.112248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:05.112386 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.112492 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.112655 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":119,"threads_started":1}
I20260810 04:19:05.113492 398 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41741
I20260810 04:19:05.113579 398 heartbeater.cc:464] Registering TS with master...
I20260810 04:19:05.113811 398 heartbeater.cc:511] Master 127.15.216.126:41741 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
I20260810 04:19:05.114447 32658 ts_manager.cc:194] Registered new tserver with Master: e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293)
I20260810 04:19:05.115095 32658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:33011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:19:05.132833 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41741
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:05.218677 403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:05.218873 403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:05.218896 403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:05.219961 403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:05.219995 403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:05.220021 403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:19:05.221395 403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:05.221575 403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:05.221827 403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:05.226914 409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:05.226924 411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:05.226922 408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:05.227272 403 server_base.cc:1034] running on GCE node
I20260810 04:19:05.227473 403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:05.227722 403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:05.228868 403 hybrid_clock.cc:648] HybridClock initialized: now 1786335545228855 us; error 28 us; skew 500 ppm
Aug 10 04:19:05 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:05.231556 403 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:19:05.232051 403 webserver.cc:460] Webserver started at http://127.15.216.66:46471/ using document root <none> and password file <none>
I20260810 04:19:05.232302 403 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:05.232372 403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:05.232530 403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:19:05.233675 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "61ea357ec632432e89661aae77d70395"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "e8d28ced4137e160b2f0947023f3a6be"
server_key_iv: "4787a3996b96de2afa7fbb3a3184276e"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.234064 403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "61ea357ec632432e89661aae77d70395"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "e8d28ced4137e160b2f0947023f3a6be"
server_key_iv: "4787a3996b96de2afa7fbb3a3184276e"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.235438 403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:19:05.236213 419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.236459 403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:19:05.236554 403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "61ea357ec632432e89661aae77d70395"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "e8d28ced4137e160b2f0947023f3a6be"
server_key_iv: "4787a3996b96de2afa7fbb3a3184276e"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.236637 403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:05.254691 403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:05.257225 403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:05.257448 403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:05.257789 403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:05.258183 403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:19:05.258239 403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.258270 403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:19:05.258287 403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.265765 403 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:41665
I20260810 04:19:05.265805 541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:41665 every 8 connection(s)
I20260810 04:19:05.266222 403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:19:05.268757 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 403
I20260810 04:19:05.268850 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:19:05.269057 16225 external_mini_cluster.cc:1428] Setting key c2f8a6c76b1dcb4a98dabe5a09d98c94
Aug 10 04:19:05 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:19:05.277347 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.267110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:48267 (local address 127.15.216.126:41741)
0810 04:19:05.267166 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.267168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.267448 (+ 280us) server_negotiation.cc:407] Connection header received
0810 04:19:05.268015 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.268016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.268026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.268071 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:05.269337 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.270058 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.271079 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.271299 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.271387 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.274016 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:05.274018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:05.274018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:05.274026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:05.275845 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.276267 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.276269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.276269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.276339 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.276592 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.276593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.276593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.276758 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:05.276852 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.277113 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.277211 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:19:05.277362 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.267076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:48267)
0810 04:19:05.267296 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.267348 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.267902 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.268147 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.268150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.268899 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:05.269200 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.269207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.270170 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.270171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.270979 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.270982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.271404 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.271405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.271536 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.272123 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:05.272135 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:05.273883 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:05.275951 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:05.275958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:05.275960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:05.276183 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:05.276419 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:05.276420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:05.276420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:05.276522 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:05.276843 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:05.276847 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:05.276975 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.277060 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.277191 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
I20260810 04:19:05.277925 542 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41741
I20260810 04:19:05.278044 542 heartbeater.cc:464] Registering TS with master...
I20260810 04:19:05.278227 542 heartbeater.cc:511] Master 127.15.216.126:41741 requested a full tablet report, sending...
I20260810 04:19:05.278651 32658 ts_manager.cc:194] Registered new tserver with Master: 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665)
I20260810 04:19:05.279094 32658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:48267
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:19:05.298863 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41741
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:05.389029 547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:05.389276 547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:05.389315 547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:05.390483 547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:05.390555 547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:05.390604 547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:19:05.392079 547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:05.392361 547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:05.392619 547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:05.397940 553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:05.397958 555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:05.398097 552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:05.398337 547 server_base.cc:1034] running on GCE node
I20260810 04:19:05.398543 547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:05.398802 547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:05.399986 547 hybrid_clock.cc:648] HybridClock initialized: now 1786335545399975 us; error 37 us; skew 500 ppm
Aug 10 04:19:05 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:05.402665 547 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:19:05.403158 547 webserver.cc:460] Webserver started at http://127.15.216.67:40909/ using document root <none> and password file <none>
I20260810 04:19:05.403393 547 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:05.403462 547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:05.403615 547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:19:05.404786 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "1df1c60e8f844f8584f63177d28f5a1f"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "c2e77c52e9ef3ca1e38af7aedccda69d"
server_key_iv: "03269c382035b7bc756359e55c757776"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.405153 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "1df1c60e8f844f8584f63177d28f5a1f"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "c2e77c52e9ef3ca1e38af7aedccda69d"
server_key_iv: "03269c382035b7bc756359e55c757776"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.406666 547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:19:05.407500 563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.407680 547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:19:05.407794 547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "1df1c60e8f844f8584f63177d28f5a1f"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "c2e77c52e9ef3ca1e38af7aedccda69d"
server_key_iv: "03269c382035b7bc756359e55c757776"
server_key_version: "encryptionkey@0"
I20260810 04:19:05.407869 547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:05.422158 547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:05.424453 547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:05.424619 547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:05.424886 547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:05.425159 547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:19:05.425191 547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.425213 547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:19:05.425226 547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:05.431317 547 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:34923
I20260810 04:19:05.431396 684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:34923 every 8 connection(s)
I20260810 04:19:05.431744 547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:19:05.435009 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 547
I20260810 04:19:05.435129 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:19:05.435315 16225 external_mini_cluster.cc:1428] Setting key e8cd5678c3c5168bc9a0dd84f6e78cb7
Aug 10 04:19:05 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335545, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:19:05.441686 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.432607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:34107 (local address 127.15.216.126:41741)
0810 04:19:05.432673 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.432675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.432887 (+ 212us) server_negotiation.cc:407] Connection header received
0810 04:19:05.433461 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.433462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.433476 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.433532 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:05.434599 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.435113 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.436419 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.436639 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.436722 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.438995 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:05.438997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:05.438997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:05.439004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:05.440386 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.440807 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.440808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.440808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.440866 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.441095 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.441095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.441096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.441202 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:05.441252 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.441494 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.441561 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 04:19:05.441864 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.432588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:34107)
0810 04:19:05.432762 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.432803 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.433333 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.433636 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.433640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.434277 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:05.434481 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.434487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.435227 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.435228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.436156 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.436160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.436733 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.436734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.436881 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.437389 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:05.437398 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:05.438893 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:05.440465 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:05.440472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:05.440474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:05.440739 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:05.440935 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:05.440936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:05.440936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:05.441040 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:05.441257 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:05.441261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:05.441379 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.441467 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.441648 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260810 04:19:05.442346 685 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41741
I20260810 04:19:05.442468 685 heartbeater.cc:464] Registering TS with master...
I20260810 04:19:05.442653 685 heartbeater.cc:511] Master 127.15.216.126:41741 requested a full tablet report, sending...
I20260810 04:19:05.443013 32658 ts_manager.cc:194] Registered new tserver with Master: 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
I20260810 04:19:05.443352 32658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:34107
I20260810 04:19:05.447086 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:19:05.454180 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.448449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.15.216.126:41741)
0810 04:19:05.448508 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.448511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.448595 (+ 84us) server_negotiation.cc:407] Connection header received
0810 04:19:05.448642 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.448642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.448650 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.448700 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:05.449212 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.449735 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.450421 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.450607 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.450660 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.451748 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:05.451750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:05.451750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:05.451756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:05.453162 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.453467 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.453467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.453468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.453517 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:05.453736 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:05.453737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:05.453737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:05.453893 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:05.453972 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.454061 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.454118 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:19:05.455875 32658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:19:05.457679 32658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:19:05.464735 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.460396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:38293 (local address 127.0.0.1:54160)
0810 04:19:05.460561 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.460605 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.460668 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.461172 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.461176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.461178 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:05.461250 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.461253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.462687 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.462688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.463746 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.463748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.464419 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.464420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.464503 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.464516 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.464591 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.464653 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":54,"threads_started":1}
I20260810 04:19:05.464761 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.460556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.15.216.65:38293)
0810 04:19:05.460977 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.460982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.460995 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:19:05.461039 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.461041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.461059 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.461339 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:05.461414 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.462576 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.463840 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.464330 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.464419 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.464481 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.464613 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.464659 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":116,"threads_started":1}
I20260810 04:19:05.464967 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.460596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34923 (local address 127.0.0.1:35694)
0810 04:19:05.460974 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.460985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.461055 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.461367 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.461368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.461370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:05.461403 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.461404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.462766 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.462767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.463742 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.463745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.464687 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.464687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.464762 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.464765 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.464826 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.464890 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":10,"thread_start_us":66,"threads_started":1}
I20260810 04:19:05.464977 699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.460410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41665 (local address 127.0.0.1:38668)
0810 04:19:05.460711 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.460720 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.460759 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.461197 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.461198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.461200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:05.461273 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.461275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.462335 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.462336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.463563 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.463565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.464507 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.464508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.464579 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.464584 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.464615 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.464659 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":93,"threads_started":1}
I20260810 04:19:05.465180 702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.460708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35694 (local address 127.15.216.67:34923)
0810 04:19:05.460943 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.460948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.461126 (+ 178us) server_negotiation.cc:407] Connection header received
0810 04:19:05.461181 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.461183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.461200 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.461273 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:05.461489 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.462659 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.463861 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.464539 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.464599 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.464653 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.465053 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.465098 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":133,"threads_started":1}
I20260810 04:19:05.465261 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.460793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.15.216.66:41665)
0810 04:19:05.460979 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.460982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.460992 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:19:05.461055 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.461057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.461075 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.461364 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:05.461446 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.462216 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.463657 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.464401 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.465022 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.465096 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.465121 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.465165 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260810 04:19:05.466015 471 tablet_service.cc:1467] Processing CreateTablet for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde (DEFAULT_TABLE table=test-table [id=05a4c8b36ff94f809ba82246c08ae5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:19:05.466028 325 tablet_service.cc:1467] Processing CreateTablet for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde (DEFAULT_TABLE table=test-table [id=05a4c8b36ff94f809ba82246c08ae5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:19:05.466028 614 tablet_service.cc:1467] Processing CreateTablet for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde (DEFAULT_TABLE table=test-table [id=05a4c8b36ff94f809ba82246c08ae5a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:19:05.466320 471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:19:05.466358 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:19:05.466358 325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:19:05.468765 704 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Bootstrap starting.
I20260810 04:19:05.469383 704 tablet_bootstrap.cc:654] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Neither blocks nor log segments found. Creating new log.
I20260810 04:19:05.469699 706 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Bootstrap starting.
I20260810 04:19:05.470052 704 log.cc:824] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:05.470400 706 tablet_bootstrap.cc:654] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Neither blocks nor log segments found. Creating new log.
I20260810 04:19:05.470441 705 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Bootstrap starting.
I20260810 04:19:05.470914 704 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: No bootstrap required, opened a new log
I20260810 04:19:05.470978 706 log.cc:824] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:05.470995 704 ts_tablet_manager.cc:1397] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:19:05.471745 706 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: No bootstrap required, opened a new log
I20260810 04:19:05.471819 706 ts_tablet_manager.cc:1397] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:19:05.473073 704 raft_consensus.cc:348] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.473258 704 raft_consensus.cc:374] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:19:05.473299 704 raft_consensus.cc:729] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ea357ec632432e89661aae77d70395, State: Initialized, Role: FOLLOWER
I20260810 04:19:05.473434 704 consensus_queue.cc:260] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.473852 704 ts_tablet_manager.cc:1428] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 04:19:05.473884 542 heartbeater.cc:503] Master 127.15.216.126:41741 was elected leader, sending a full tablet report...
I20260810 04:19:05.473897 706 raft_consensus.cc:348] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.474076 706 raft_consensus.cc:374] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:19:05.474123 706 raft_consensus.cc:729] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dc229a78bd4717af6c964f9fa94300, State: Initialized, Role: FOLLOWER
I20260810 04:19:05.474396 706 consensus_queue.cc:260] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.474454 705 tablet_bootstrap.cc:654] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Neither blocks nor log segments found. Creating new log.
I20260810 04:19:05.474689 706 ts_tablet_manager.cc:1428] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 04:19:05.474733 398 heartbeater.cc:503] Master 127.15.216.126:41741 was elected leader, sending a full tablet report...
I20260810 04:19:05.475123 705 log.cc:824] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:05.476027 705 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: No bootstrap required, opened a new log
I20260810 04:19:05.476110 705 ts_tablet_manager.cc:1397] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 04:19:05.477757 705 raft_consensus.cc:348] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.477910 705 raft_consensus.cc:374] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:19:05.477957 705 raft_consensus.cc:729] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df1c60e8f844f8584f63177d28f5a1f, State: Initialized, Role: FOLLOWER
I20260810 04:19:05.478088 705 consensus_queue.cc:260] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.478375 705 ts_tablet_manager.cc:1428] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:19:05.478462 685 heartbeater.cc:503] Master 127.15.216.126:41741 was elected leader, sending a full tablet report...
W20260810 04:19:05.516923 543 tablet.cc:2369] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:19:05.586134 710 raft_consensus.cc:482] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:05.586355 710 raft_consensus.cc:504] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.586786 710 leader_election.cc:288] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
I20260810 04:19:05.589867 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.587139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:50753 (local address 127.15.216.65:38293)
0810 04:19:05.587227 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.587229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.587240 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:19:05.587280 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.587281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.587297 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.587384 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:05.587574 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.588112 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.589082 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.589442 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.589526 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.589535 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.589760 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.589799 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 04:19:05.589865 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.587000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:38293 (local address 127.15.216.66:50753)
0810 04:19:05.587073 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.587086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.587153 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.587399 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.587400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.587402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:05.587479 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.587481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.588204 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.588205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.588975 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.588978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.589524 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.589525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.589665 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.589668 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.589737 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.589784 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":24}
I20260810 04:19:05.589988 702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.587268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:59793 (local address 127.15.216.67:34923)
0810 04:19:05.587349 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.587351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.587414 (+ 63us) server_negotiation.cc:407] Connection header received
0810 04:19:05.587489 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.587490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.587499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.587572 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:05.587761 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.588222 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.589162 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.589571 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.589692 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.589700 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.589887 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.589927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 04:19:05.589998 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.587137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34923 (local address 127.15.216.66:59793)
0810 04:19:05.587348 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.587359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:05.587422 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:05.587578 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:05.587579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:05.587580 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:05.587680 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.587682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.588313 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.588314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.589069 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:05.589071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.589699 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.589700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:05.589780 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.589784 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:19:05.589873 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 04:19:05.589914 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260810 04:19:05.590255 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cdf0ff24d34602b3ea9c4a5c3a3bde" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dc229a78bd4717af6c964f9fa94300" is_pre_election: true
I20260810 04:19:05.590402 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cdf0ff24d34602b3ea9c4a5c3a3bde" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df1c60e8f844f8584f63177d28f5a1f" is_pre_election: true
I20260810 04:19:05.590459 346 raft_consensus.cc:2454] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 0.
I20260810 04:19:05.590545 634 raft_consensus.cc:2454] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 0.
I20260810 04:19:05.590700 421 leader_election.cc:302] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61ea357ec632432e89661aae77d70395, e3dc229a78bd4717af6c964f9fa94300; no voters:
I20260810 04:19:05.590842 710 raft_consensus.cc:2781] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:19:05.590900 710 raft_consensus.cc:482] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:19:05.590925 710 raft_consensus.cc:3037] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:19:05.591632 710 raft_consensus.cc:504] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.591777 710 leader_election.cc:288] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 election: Requested vote from peers e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
I20260810 04:19:05.591969 634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cdf0ff24d34602b3ea9c4a5c3a3bde" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df1c60e8f844f8584f63177d28f5a1f"
I20260810 04:19:05.592051 634 raft_consensus.cc:3037] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:19:05.592022 346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cdf0ff24d34602b3ea9c4a5c3a3bde" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dc229a78bd4717af6c964f9fa94300"
I20260810 04:19:05.592131 346 raft_consensus.cc:3037] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:19:05.592801 634 raft_consensus.cc:2454] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 1.
I20260810 04:19:05.592967 421 leader_election.cc:302] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395; no voters:
I20260810 04:19:05.593000 346 raft_consensus.cc:2454] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 1.
I20260810 04:19:05.593068 710 raft_consensus.cc:2781] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:19:05.593263 710 raft_consensus.cc:686] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 1 LEADER]: Becoming Leader. State: Replica: 61ea357ec632432e89661aae77d70395, State: Running, Role: LEADER
I20260810 04:19:05.593381 710 consensus_queue.cc:237] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:05.594340 32658 catalog_manager.cc:5495] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61ea357ec632432e89661aae77d70395 (127.15.216.66). New cstate: current_term: 1 leader_uuid: "61ea357ec632432e89661aae77d70395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } health_report { overall_health: UNKNOWN } } }
W20260810 04:19:05.601907 399 tablet.cc:2369] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:19:05.614846 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.612302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.15.216.66:41665)
0810 04:19:05.612369 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:05.612372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:05.612384 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:19:05.612454 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:05.612455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:05.612476 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:05.612556 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:19:05.612673 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.613147 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.613994 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:05.614192 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:05.614279 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:05.614467 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:19:05.614573 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:19:05.614666 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:05.614710 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 04:19:05.614732 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:19:05.616937 634 raft_consensus.cc:1261] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Refusing update from remote peer 61ea357ec632432e89661aae77d70395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:19:05.616937 346 raft_consensus.cc:1261] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Refusing update from remote peer 61ea357ec632432e89661aae77d70395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:19:05.617318 710 consensus_queue.cc:1035] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:19:05.617543 715 consensus_queue.cc:1035] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:19:05.620107 720 mvcc.cc:205] Tried to move back new op lower bound from 7316830394843496448 to 7316830394751066112. Current Snapshot: MvccSnapshot[applied={T|T < 7316830394843496448}]
I20260810 04:19:05.620596 721 mvcc.cc:205] Tried to move back new op lower bound from 7316830394843496448 to 7316830394751066112. Current Snapshot: MvccSnapshot[applied={T|T < 7316830394843496448}]
I20260810 04:19:05.622506 32657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:19:05.622613 32657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:19:05.623488 32657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Sending DeleteTablet for 3 replicas of tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde
I20260810 04:19:05.623849 325 tablet_service.cc:1514] Processing DeleteTablet for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:19:05 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:54160
I20260810 04:19:05.623875 614 tablet_service.cc:1514] Processing DeleteTablet for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:19:05 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:35694
I20260810 04:19:05.624150 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 32724
I20260810 04:19:05.624147 471 tablet_service.cc:1514] Processing DeleteTablet for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:19:05 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:38668
I20260810 04:19:05.626607 731 tablet_replica.cc:330] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: stopping tablet replica
I20260810 04:19:05.626713 731 raft_consensus.cc:2229] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:19:05.626821 731 raft_consensus.cc:2258] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:19:05.627455 731 ts_tablet_manager.cc:1905] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:19:05.628789 732 tablet_replica.cc:330] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: stopping tablet replica
I20260810 04:19:05.628882 732 raft_consensus.cc:2229] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:19:05.628985 732 raft_consensus.cc:2258] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:19:05.629503 732 ts_tablet_manager.cc:1905] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:19:05.629531 731 ts_tablet_manager.cc:1918] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:19:05.629617 731 log.cc:1196] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/d4cdf0ff24d34602b3ea9c4a5c3a3bde
I20260810 04:19:05.629910 731 ts_tablet_manager.cc:1939] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 1df1c60e8f844f8584f63177d28f5a1f: Deleting consensus metadata
I20260810 04:19:05.630326 32643 catalog_manager.cc:4841] TS 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde (table test-table [id=05a4c8b36ff94f809ba82246c08ae5a5]) successfully deleted
I20260810 04:19:05.631312 732 ts_tablet_manager.cc:1918] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:19:05.631366 732 log.cc:1196] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/d4cdf0ff24d34602b3ea9c4a5c3a3bde
I20260810 04:19:05.631573 732 ts_tablet_manager.cc:1939] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P 61ea357ec632432e89661aae77d70395: Deleting consensus metadata
I20260810 04:19:05.631928 32645 catalog_manager.cc:4841] TS 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde (table test-table [id=05a4c8b36ff94f809ba82246c08ae5a5]) successfully deleted
W20260810 04:19:05.638120 32643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:38293 (error 108)
I20260810 04:19:05.638285 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 403
I20260810 04:19:05.638685 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:05.638569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:38293 (local address 127.0.0.1:54170)
0810 04:19:05.638620 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:05.638652 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:19:05.638782 32643 catalog_manager.cc:4584] TS e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde: Network error: Client connection negotiation failed: client connection to 127.15.216.65:38293: connect: Connection refused (error 111)
I20260810 04:19:05.645514 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 547
I20260810 04:19:05.653261 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 32623
I20260810 04:19:25.661036 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41741
--webserver_interface=127.15.216.126
--webserver_port=38899
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:25.751360 734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:25.751595 734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:25.751619 734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:25.752784 734 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:19:25.752840 734 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:19:25.752853 734 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:19:25.752866 734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:19:25.752970 734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:25.753034 734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:25.753050 734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:19:25.753064 734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:19:25.754866 734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=38899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:25.755116 734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:25.755309 734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:25.766456 740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:25.766466 734 server_base.cc:1034] running on GCE node
W20260810 04:19:25.766426 744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:25.766426 741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:25.767058 734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:25.767462 734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:25.768720 734 hybrid_clock.cc:648] HybridClock initialized: now 1786335565768693 us; error 73 us; skew 500 ppm
Aug 10 04:19:25 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335565, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:25.771718 734 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:19:25.772262 734 webserver.cc:460] Webserver started at http://127.15.216.126:38899/ using document root <none> and password file <none>
I20260810 04:19:25.772483 734 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:25.772548 734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:25.774173 734 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:19:25.775074 751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:25.775329 734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:19:25.775432 734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e5ef2b69afaa4404b4214f25d8b06087"
format_stamp: "Formatted at 2026-08-10 04:19:04 on dist-test-slave-g350"
server_key: "800d42379678694ba01af291e7c5a1ef"
server_key_iv: "fbdb6846ada38ba5fe0738c21dd5dc59"
server_key_version: "encryptionkey@0"
I20260810 04:19:25.775875 734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:25.790669 734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:25.792907 734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:25.793035 734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:25.798666 734 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41741
I20260810 04:19:25.798738 807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41741 every 8 connection(s)
I20260810 04:19:25.799216 734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:19:25.801118 808 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:19:25.801889 808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Bootstrap starting.
I20260810 04:19:25.808882 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 734
I20260810 04:19:25.809293 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:38293
--local_ip_for_outbound_sockets=127.15.216.65
--tserver_master_addrs=127.15.216.126:41741
--webserver_port=38161
--webserver_interface=127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:19:25.812397 808 log.cc:824] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:25.814687 808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:19:25.815058 808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Bootstrap complete.
I20260810 04:19:25.818013 808 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:25.818424 808 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ef2b69afaa4404b4214f25d8b06087, State: Initialized, Role: FOLLOWER
I20260810 04:19:25.818607 808 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:25.818686 808 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:19:25.818738 808 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:19:25.818779 808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 1 FOLLOWER]: Advancing to term 2
I20260810 04:19:25.820178 808 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:25.820317 808 leader_election.cc:302] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ef2b69afaa4404b4214f25d8b06087; no voters:
I20260810 04:19:25.820559 808 leader_election.cc:288] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 04:19:25.820613 813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 04:19:25.820762 813 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 LEADER]: Becoming Leader. State: Replica: e5ef2b69afaa4404b4214f25d8b06087, State: Running, Role: LEADER
I20260810 04:19:25.820858 813 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:25.820868 808 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:19:25.821359 814 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5ef2b69afaa4404b4214f25d8b06087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } } }
I20260810 04:19:25.821491 814 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: This master's current role is: LEADER
I20260810 04:19:25.821833 819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:19:25.822736 815 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ef2b69afaa4404b4214f25d8b06087. Latest consensus state: current_term: 2 leader_uuid: "e5ef2b69afaa4404b4214f25d8b06087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } } }
I20260810 04:19:25.822820 815 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: This master's current role is: LEADER
I20260810 04:19:25.822872 819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:19:25.823007 819 catalog_manager.cc:1259] Loaded cluster ID: dec24dcbad3242699f8e9cab514a1d7c
I20260810 04:19:25.823051 819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:19:25.824638 819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:19:25.824851 819 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Loaded TSK: 0
I20260810 04:19:25.846011 819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Generated new TSK 1
I20260810 04:19:25.846143 819 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:19:25.912452 810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:25.912676 810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:25.912703 810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:25.913965 810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:25.914011 810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:25.914038 810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:19:25.915604 810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:38293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=38161
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:25.915868 810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:25.916162 810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:25.923082 839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:25.923120 840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:25.923379 810 server_base.cc:1034] running on GCE node
W20260810 04:19:25.923123 842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:25.923760 810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:25.924057 810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:25.925261 810 hybrid_clock.cc:648] HybridClock initialized: now 1786335565925243 us; error 34 us; skew 500 ppm
Aug 10 04:19:25 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335565, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:25.928215 810 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:19:25.928776 810 webserver.cc:460] Webserver started at http://127.15.216.65:38161/ using document root <none> and password file <none>
I20260810 04:19:25.929039 810 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:25.929134 810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:25.930781 810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:25.931720 851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:25.931970 810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:25.932092 810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "e3dc229a78bd4717af6c964f9fa94300"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "8cb5e653b4f24dd3e873698994f5b82c"
server_key_iv: "0ee9e437ab93174de9c4d6affa745cce"
server_key_version: "encryptionkey@0"
I20260810 04:19:25.932533 810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:25.946171 810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:25.948990 810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:25.949183 810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:25.949457 810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:25.949998 859 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:19:25.951973 810 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:19:25.952062 810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:19:25.952112 810 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:19:25.952862 810 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:19:25.952926 810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:19:25.952999 859 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Bootstrap starting.
I20260810 04:19:25.962984 810 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:38293
I20260810 04:19:25.963406 810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:19:25.966476 977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:38293 every 8 connection(s)
I20260810 04:19:25.967391 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 810
I20260810 04:19:25.967686 859 log.cc:824] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:25.967711 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:41665
--local_ip_for_outbound_sockets=127.15.216.66
--tserver_master_addrs=127.15.216.126:41741
--webserver_port=46471
--webserver_interface=127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:19:25.970775 859 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:19:25.971163 859 tablet_bootstrap.cc:492] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Bootstrap complete.
I20260810 04:19:25.972579 859 ts_tablet_manager.cc:1397] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
Aug 10 04:19:25 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335565, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:19:25.973920 859 raft_consensus.cc:348] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:25.974258 859 raft_consensus.cc:729] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dc229a78bd4717af6c964f9fa94300, State: Initialized, Role: FOLLOWER
I20260810 04:19:25.974407 859 consensus_queue.cc:260] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } } peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } }
I20260810 04:19:25.974750 859 ts_tablet_manager.cc:1428] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260810 04:19:25.977169 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:25.964267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:33125 (local address 127.15.216.126:41741)
0810 04:19:25.965806 (+ 1539us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:25.965814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:25.965850 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:19:25.965964 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:25.965969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:25.966011 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:25.966172 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:25.967138 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.968499 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.970181 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.970603 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.970737 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:25.973193 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:25.973199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:25.973200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:25.973217 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:25.975261 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:25.975735 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:25.975739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:25.975739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:25.975852 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:25.976150 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:25.976151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:25.976151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:25.976407 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:25.976504 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:25.976834 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 04:19:25.976968 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1474,"thread_start_us":106,"threads_started":1}
I20260810 04:19:25.977169 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:25.964181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:33125)
0810 04:19:25.964386 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:25.964431 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:25.964935 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:25.966220 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:25.966225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:25.966775 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:25.966997 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:25.967003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.968674 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.968676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:25.970071 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:25.970074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.970731 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.970732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:25.970895 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:25.971561 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:25.971570 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:25.973061 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:25.975371 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:25.975379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:25.975381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:25.975625 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:25.975942 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:25.975943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:25.975943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:25.976067 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:25.976494 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:25.976502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:25.976645 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 04:19:25.976754 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 04:19:25.976959 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":130,"threads_started":1}
I20260810 04:19:25.979076 978 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41741
I20260810 04:19:25.979241 978 heartbeater.cc:464] Registering TS with master...
I20260810 04:19:25.979493 978 heartbeater.cc:511] Master 127.15.216.126:41741 requested a full tablet report, sending...
I20260810 04:19:25.980552 769 ts_manager.cc:194] Registered new tserver with Master: e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293)
I20260810 04:19:25.982023 769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:33125
I20260810 04:19:25.983146 978 heartbeater.cc:503] Master 127.15.216.126:41741 was elected leader, sending a full tablet report...
Aug 10 04:19:25 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335565, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.65@KRBTEST.COM
I20260810 04:19:25.991823 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:25.981443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:38293 (local address 127.0.0.1:59946)
0810 04:19:25.981694 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:25.981750 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:25.981837 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:25.982355 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:25.982362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:25.982791 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:25.982871 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:25.982875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.984177 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.984178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:25.985240 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:25.985244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.986114 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.986116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:25.986239 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:25.986829 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:25.986836 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:25.988484 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:25.990477 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:25.990486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:25.990488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:25.990715 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:25.990967 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:25.990968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:25.990969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:25.991042 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:25.991381 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:25.991389 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:25.991452 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 04:19:25.991560 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 04:19:25.991713 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260810 04:19:25.991823 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:25.981790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.15.216.65:38293)
0810 04:19:25.982014 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:25.982019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:25.982032 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:19:25.982081 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:25.982083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:25.982102 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:25.982320 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:25.982987 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.984074 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.985444 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:25.985910 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:25.985990 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:25.988601 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:25.988612 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:25.988613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:25.988621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:25.990346 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:25.990819 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:25.990823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:25.990825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:25.990891 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:25.991120 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:25.991121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:25.991121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:25.991293 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:25.991407 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:25.991602 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 04:19:25.991713 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260810 04:19:25.992288 905 tablet_service.cc:1514] Processing DeleteTablet for tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:19:05 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:59946
I20260810 04:19:25.992864 987 tablet_replica.cc:330] stopping tablet replica
I20260810 04:19:25.992992 987 raft_consensus.cc:2229] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:19:25.993095 987 raft_consensus.cc:2258] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:19:25.993443 987 ts_tablet_manager.cc:1905] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:19:25.995643 987 ts_tablet_manager.cc:1918] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:19:25.995718 987 log.cc:1196] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal/wals/d4cdf0ff24d34602b3ea9c4a5c3a3bde
I20260810 04:19:25.996001 987 ts_tablet_manager.cc:1939] T d4cdf0ff24d34602b3ea9c4a5c3a3bde P e3dc229a78bd4717af6c964f9fa94300: Deleting consensus metadata
I20260810 04:19:25.996613 753 catalog_manager.cc:4841] TS e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293): tablet d4cdf0ff24d34602b3ea9c4a5c3a3bde (table test-table [id=05a4c8b36ff94f809ba82246c08ae5a5]) successfully deleted
W20260810 04:19:26.072317 983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:26.072628 983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:26.072677 983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:26.074271 983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:26.074401 983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:26.074463 983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:19:26.076550 983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:41665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=46471
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:26.076920 983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:26.077196 983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:26.083341 993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:26.083540 983 server_base.cc:1034] running on GCE node
W20260810 04:19:26.083427 992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:26.083323 995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:26.083870 983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:26.084128 983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:26.085358 983 hybrid_clock.cc:648] HybridClock initialized: now 1786335566085307 us; error 46 us; skew 500 ppm
Aug 10 04:19:26 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335566, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:26.088256 983 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:19:26.088826 983 webserver.cc:460] Webserver started at http://127.15.216.66:46471/ using document root <none> and password file <none>
I20260810 04:19:26.089102 983 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:26.089200 983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:26.090770 983 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:26.091480 1002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:26.091662 983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:26.091776 983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "61ea357ec632432e89661aae77d70395"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "e8d28ced4137e160b2f0947023f3a6be"
server_key_iv: "4787a3996b96de2afa7fbb3a3184276e"
server_key_version: "encryptionkey@0"
I20260810 04:19:26.092156 983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:26.102965 983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:26.105242 983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:26.105451 983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:26.105810 983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:26.106172 983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:19:26.106237 983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:26.106289 983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:19:26.106318 983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:26.113124 983 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:41665
I20260810 04:19:26.113173 1125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:41665 every 8 connection(s)
I20260810 04:19:26.113690 983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:19:26 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335566, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:19:26.120555 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 983
I20260810 04:19:26.120815 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:34923
--local_ip_for_outbound_sockets=127.15.216.67
--tserver_master_addrs=127.15.216.126:41741
--webserver_port=40909
--webserver_interface=127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:19:26.124903 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.114536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:33023 (local address 127.15.216.126:41741)
0810 04:19:26.114601 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.114603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.114772 (+ 169us) server_negotiation.cc:407] Connection header received
0810 04:19:26.115291 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.115292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.115307 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.115371 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:26.116400 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.116937 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.118142 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.118353 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.118417 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.121097 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:26.121099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:26.121099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:26.121108 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:26.123130 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.123745 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.123747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.123747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.123808 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.124126 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.124126 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.124127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.124248 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:26.124304 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.124673 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.124806 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:19:26.125106 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.114455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:33023)
0810 04:19:26.114631 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:26.114677 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:26.115177 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:26.115427 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:26.115431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:26.116055 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:26.116277 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.116282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.117042 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.117043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.118059 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.118062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.118452 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.118452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.118586 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.119165 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:26.119174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:26.120954 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:26.123224 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:26.123242 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:26.123245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:26.123648 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:26.123887 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:26.123887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:26.123888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:26.124049 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:26.124319 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:26.124327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:26.124491 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 04:19:26.124641 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 04:19:26.124838 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260810 04:19:26.125689 1127 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41741
I20260810 04:19:26.125833 1127 heartbeater.cc:464] Registering TS with master...
I20260810 04:19:26.126066 1127 heartbeater.cc:511] Master 127.15.216.126:41741 requested a full tablet report, sending...
I20260810 04:19:26.126569 769 ts_manager.cc:194] Registered new tserver with Master: 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665)
I20260810 04:19:26.127223 769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:33023
W20260810 04:19:26.211230 1130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:26.211488 1130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:26.211516 1130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:26.212667 1130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:26.212711 1130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:26.212738 1130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:19:26.214226 1130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:34923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=40909
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:26.214457 1130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:26.214666 1130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:26.220623 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:26.220636 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:26.220640 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:26.221038 1130 server_base.cc:1034] running on GCE node
I20260810 04:19:26.221351 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:26.221818 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:26.223006 1130 hybrid_clock.cc:648] HybridClock initialized: now 1786335566222992 us; error 30 us; skew 500 ppm
Aug 10 04:19:26 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335566, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:26.226101 1130 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:19:26.226753 1130 webserver.cc:460] Webserver started at http://127.15.216.67:40909/ using document root <none> and password file <none>
I20260810 04:19:26.227049 1130 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:26.227149 1130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:26.228858 1130 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:19:26.229732 1146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:26.229993 1130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:26.230113 1130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "1df1c60e8f844f8584f63177d28f5a1f"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "c2e77c52e9ef3ca1e38af7aedccda69d"
server_key_iv: "03269c382035b7bc756359e55c757776"
server_key_version: "encryptionkey@0"
I20260810 04:19:26.230595 1130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:26.245781 1130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:26.248440 1130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:26.248624 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:26.248919 1130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:26.249293 1130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:19:26.249352 1130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:26.249389 1130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:19:26.249435 1130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:26.256448 1130 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:34923
I20260810 04:19:26.256500 1268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:34923 every 8 connection(s)
I20260810 04:19:26.256876 1130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:19:26.259999 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 1130
Aug 10 04:19:26 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335566, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:19:26.268368 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.260569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.15.216.126:41741)
0810 04:19:26.260728 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.260730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.260740 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:19:26.260788 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.260789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.260801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.260872 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:26.261524 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.262204 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.263041 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.263330 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.263422 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.264950 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:26.264951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:26.264951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:26.264959 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:26.266892 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.267342 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.267343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.267344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.267401 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.267679 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.267681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.267681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.267856 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:26.268076 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.268125 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.268236 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260810 04:19:26.268638 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.257703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:59221)
0810 04:19:26.257897 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:26.257977 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:26.258552 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:26.258821 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:26.258825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:26.259450 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:26.259661 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.259666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.260544 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.260545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.261632 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.261635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.262040 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.262041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.262200 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.262827 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:26.262836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:26.264499 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:26.266644 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:26.266656 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:26.266659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:26.267013 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:26.267238 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:26.267239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:26.267239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:26.267453 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:26.267774 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:26.267783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:26.267989 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 04:19:26.268125 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 04:19:26.268326 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260810 04:19:26.268941 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.257901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:59221 (local address 127.15.216.126:41741)
0810 04:19:26.257974 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.257976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.258072 (+ 96us) server_negotiation.cc:407] Connection header received
0810 04:19:26.258674 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.258675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.258689 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.258747 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:26.259770 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.260405 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.261716 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.261943 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.262001 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.264603 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:26.264605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:26.264605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:26.264614 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:26.266542 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.267101 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.267102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.267103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.267160 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.267529 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.267531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.267531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.267690 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:26.267742 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.268181 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.268387 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":43}
I20260810 04:19:26.269318 1269 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41741
I20260810 04:19:26.269415 1269 heartbeater.cc:464] Registering TS with master...
I20260810 04:19:26.269568 1269 heartbeater.cc:511] Master 127.15.216.126:41741 requested a full tablet report, sending...
I20260810 04:19:26.270061 768 ts_manager.cc:194] Registered new tserver with Master: 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
I20260810 04:19:26.270406 768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:59221
I20260810 04:19:26.280486 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:19:26.283564 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.280942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.15.216.126:41741)
0810 04:19:26.280995 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.280997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.281051 (+ 54us) server_negotiation.cc:407] Connection header received
0810 04:19:26.281126 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.281127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.281167 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.281233 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:19:26.281430 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.282011 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.282826 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.283038 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.283094 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.283299 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:19:26.283356 (+ 57us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 04:19:26.283507 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57954: token expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:19:26.283658 981 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57954: token expired
W20260810 04:19:26.283985 690 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.216.126:41741)
I20260810 04:19:26.291109 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.284432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.15.216.126:41741)
0810 04:19:26.284476 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.284478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.284485 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:19:26.284503 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.284503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.284512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.284565 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:26.285120 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.285690 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.286444 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.286626 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.286786 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.288166 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:26.288167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:26.288168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:26.288175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:26.289844 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.290243 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.290245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.290245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.290318 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:26.290575 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:26.290577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:26.290577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:26.290713 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:26.290815 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.290960 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.291033 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:19:26.292781 768 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:19:26.294754 768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:19:26.297928 905 tablet_service.cc:1467] Processing CreateTablet for tablet aadf3fefffd6421f8a15e8f0141828ac (DEFAULT_TABLE table=test-table [id=b61eb28452c84588a8c3e0eac05b566b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:19:26.298170 905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadf3fefffd6421f8a15e8f0141828ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:19:26.299727 859 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Bootstrap starting.
I20260810 04:19:26.300125 859 tablet_bootstrap.cc:654] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Neither blocks nor log segments found. Creating new log.
I20260810 04:19:26.300741 859 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: No bootstrap required, opened a new log
I20260810 04:19:26.300812 859 ts_tablet_manager.cc:1397] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:26.301175 859 raft_consensus.cc:348] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.301267 859 raft_consensus.cc:374] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:19:26.301335 859 raft_consensus.cc:729] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dc229a78bd4717af6c964f9fa94300, State: Initialized, Role: FOLLOWER
I20260810 04:19:26.301383 859 consensus_queue.cc:260] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.301472 859 ts_tablet_manager.cc:1428] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:26.302032 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.297506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34923 (local address 127.0.0.1:53896)
0810 04:19:26.297570 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:26.297583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:26.297652 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:26.298354 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:26.298356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:26.298358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:26.298420 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.298422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.299312 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.299314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.300548 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.300551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.301716 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.301718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.301798 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.301825 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 04:19:26.301858 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 04:19:26.301952 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":29}
I20260810 04:19:26.302418 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.297681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.15.216.67:34923)
0810 04:19:26.298148 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.298152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.298165 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:19:26.298230 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.298233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.298250 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.298347 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:26.298521 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.299215 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.300790 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.301580 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.302148 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.302238 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.302267 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.302315 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":131,"threads_started":1}
I20260810 04:19:26.303115 1198 tablet_service.cc:1467] Processing CreateTablet for tablet aadf3fefffd6421f8a15e8f0141828ac (DEFAULT_TABLE table=test-table [id=b61eb28452c84588a8c3e0eac05b566b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:19:26.303494 1198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadf3fefffd6421f8a15e8f0141828ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:19:26.303978 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.297554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41665 (local address 127.0.0.1:52310)
0810 04:19:26.299760 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:26.299773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:26.299828 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:26.300064 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:26.300065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:26.300068 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:26.300122 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.300123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.301757 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.301757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.302942 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.302945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.303717 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.303718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.303785 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.303789 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:19:26.303857 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 04:19:26.303902 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2189,"thread_start_us":123,"threads_started":1}
I20260810 04:19:26.303995 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.297909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.15.216.66:41665)
0810 04:19:26.298093 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.298099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.299895 (+ 1796us) server_negotiation.cc:407] Connection header received
0810 04:19:26.299948 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.299951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.299967 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.300043 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:26.300202 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.301667 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.303034 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.303627 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.303716 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.303776 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.303873 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.303912 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260810 04:19:26.304653 1053 tablet_service.cc:1467] Processing CreateTablet for tablet aadf3fefffd6421f8a15e8f0141828ac (DEFAULT_TABLE table=test-table [id=b61eb28452c84588a8c3e0eac05b566b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:19:26.305012 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aadf3fefffd6421f8a15e8f0141828ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:19:26.306710 1281 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Bootstrap starting.
I20260810 04:19:26.307365 1281 tablet_bootstrap.cc:654] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Neither blocks nor log segments found. Creating new log.
I20260810 04:19:26.308084 1281 log.cc:824] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:26.308121 1282 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Bootstrap starting.
I20260810 04:19:26.308699 1282 tablet_bootstrap.cc:654] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Neither blocks nor log segments found. Creating new log.
I20260810 04:19:26.308944 1281 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: No bootstrap required, opened a new log
I20260810 04:19:26.309037 1281 ts_tablet_manager.cc:1397] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:19:26.309271 1282 log.cc:824] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:26.310057 1282 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: No bootstrap required, opened a new log
I20260810 04:19:26.310111 1282 ts_tablet_manager.cc:1397] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:19:26.310702 1281 raft_consensus.cc:348] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.310894 1281 raft_consensus.cc:374] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:19:26.310956 1281 raft_consensus.cc:729] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df1c60e8f844f8584f63177d28f5a1f, State: Initialized, Role: FOLLOWER
I20260810 04:19:26.311111 1281 consensus_queue.cc:260] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.311434 1281 ts_tablet_manager.cc:1428] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 04:19:26.311596 1269 heartbeater.cc:503] Master 127.15.216.126:41741 was elected leader, sending a full tablet report...
I20260810 04:19:26.311939 1282 raft_consensus.cc:348] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.312121 1282 raft_consensus.cc:374] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:19:26.312150 1282 raft_consensus.cc:729] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ea357ec632432e89661aae77d70395, State: Initialized, Role: FOLLOWER
I20260810 04:19:26.312288 1282 consensus_queue.cc:260] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.312580 1282 ts_tablet_manager.cc:1428] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 04:19:26.312619 1127 heartbeater.cc:503] Master 127.15.216.126:41741 was elected leader, sending a full tablet report...
W20260810 04:19:26.364288 1128 tablet.cc:2369] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:19:26.370858 1287 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:26.371011 1287 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.371411 1287 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923), e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293)
I20260810 04:19:26.375295 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.371662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34923 (local address 127.15.216.66:56039)
0810 04:19:26.371930 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:26.371940 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:26.372028 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:26.372195 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:26.372196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:26.372198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:26.372351 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.372355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.373105 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.373106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.374192 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.374195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.374932 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.374933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.375022 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.375027 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:19:26.375111 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 04:19:26.375158 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260810 04:19:26.375248 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.371776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:56039 (local address 127.15.216.67:34923)
0810 04:19:26.371870 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.371872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.372005 (+ 133us) server_negotiation.cc:407] Connection header received
0810 04:19:26.372091 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.372092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.372103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.372183 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:26.372470 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.373015 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.374367 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.374843 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.374900 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.374908 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.375117 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.375147 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 04:19:26.375556 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.371988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:38293 (local address 127.15.216.66:55923)
0810 04:19:26.372186 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:26.372195 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:26.372299 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:26.372499 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:26.372500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:26.372502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:19:26.372573 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.372574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.373392 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.373393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.374507 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:26.374510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.375235 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.375236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:26.375316 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.375320 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:19:26.375438 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 04:19:26.375482 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":115,"threads_started":1}
I20260810 04:19:26.375576 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.372064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:55923 (local address 127.15.216.65:38293)
0810 04:19:26.372114 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.372116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.372250 (+ 134us) server_negotiation.cc:407] Connection header received
0810 04:19:26.372369 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.372370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.372402 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.372481 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:19:26.372651 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.373295 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.374603 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.375149 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.375218 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.375311 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.375454 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.375507 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:19:26.375864 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadf3fefffd6421f8a15e8f0141828ac" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df1c60e8f844f8584f63177d28f5a1f" is_pre_election: true
I20260810 04:19:26.376096 1219 raft_consensus.cc:2454] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 0.
I20260810 04:19:26.376117 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadf3fefffd6421f8a15e8f0141828ac" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dc229a78bd4717af6c964f9fa94300" is_pre_election: true
I20260810 04:19:26.376332 918 raft_consensus.cc:2454] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 0.
I20260810 04:19:26.376585 1004 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61ea357ec632432e89661aae77d70395, e3dc229a78bd4717af6c964f9fa94300; no voters:
I20260810 04:19:26.376755 1287 raft_consensus.cc:2781] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:19:26.376844 1287 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:19:26.376878 1287 raft_consensus.cc:3037] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:19:26.377697 1287 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.377868 1287 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 election: Requested vote from peers 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923), e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293)
I20260810 04:19:26.378108 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadf3fefffd6421f8a15e8f0141828ac" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df1c60e8f844f8584f63177d28f5a1f"
I20260810 04:19:26.378167 918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aadf3fefffd6421f8a15e8f0141828ac" candidate_uuid: "61ea357ec632432e89661aae77d70395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dc229a78bd4717af6c964f9fa94300"
I20260810 04:19:26.378213 1219 raft_consensus.cc:3037] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:19:26.378278 918 raft_consensus.cc:3037] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:19:26.379027 1219 raft_consensus.cc:2454] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 1.
I20260810 04:19:26.379215 1004 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395; no voters:
I20260810 04:19:26.379336 918 raft_consensus.cc:2454] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61ea357ec632432e89661aae77d70395 in term 1.
I20260810 04:19:26.379347 1287 raft_consensus.cc:2781] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:19:26.379523 1287 raft_consensus.cc:686] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 1 LEADER]: Becoming Leader. State: Replica: 61ea357ec632432e89661aae77d70395, State: Running, Role: LEADER
I20260810 04:19:26.379627 1287 consensus_queue.cc:237] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:26.380548 768 catalog_manager.cc:5495] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61ea357ec632432e89661aae77d70395 (127.15.216.66). New cstate: current_term: 1 leader_uuid: "61ea357ec632432e89661aae77d70395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } health_report { overall_health: UNKNOWN } } }
I20260810 04:19:26.397563 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.395156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.15.216.66:41665)
0810 04:19:26.395203 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:26.395205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:26.395234 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:19:26.395306 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:26.395306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:26.395327 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:26.395393 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:19:26.395519 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.396062 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.396781 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:26.396979 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:26.397038 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:26.397190 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:19:26.397289 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:19:26.397351 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:26.397453 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:19:26.397473 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:19:26.399719 1219 raft_consensus.cc:1261] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Refusing update from remote peer 61ea357ec632432e89661aae77d70395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:19:26.399719 918 raft_consensus.cc:1261] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Refusing update from remote peer 61ea357ec632432e89661aae77d70395: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:19:26.400410 1289 consensus_queue.cc:1035] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:19:26.400589 1287 consensus_queue.cc:1035] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:19:26.403942 1292 mvcc.cc:205] Tried to move back new op lower bound from 7316830479969951744 to 7316830479891521536. Current Snapshot: MvccSnapshot[applied={T|T < 7316830479969951744}]
I20260810 04:19:26.404489 1293 mvcc.cc:205] Tried to move back new op lower bound from 7316830479969951744 to 7316830479891521536. Current Snapshot: MvccSnapshot[applied={T|T < 7316830479969951744}]
I20260810 04:19:26.405896 769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:19:26.406014 769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:19:26.406850 769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Sending DeleteTablet for 3 replicas of tablet aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:26.407184 1053 tablet_service.cc:1514] Processing DeleteTablet for tablet aadf3fefffd6421f8a15e8f0141828ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:19:26 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:52310
I20260810 04:19:26.407285 1198 tablet_service.cc:1514] Processing DeleteTablet for tablet aadf3fefffd6421f8a15e8f0141828ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:19:26 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:53896
I20260810 04:19:26.407334 905 tablet_service.cc:1514] Processing DeleteTablet for tablet aadf3fefffd6421f8a15e8f0141828ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:19:26 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:59946
I20260810 04:19:26.407441 987 tablet_replica.cc:330] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: stopping tablet replica
I20260810 04:19:26.407488 987 raft_consensus.cc:2229] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:19:26.407574 987 raft_consensus.cc:2258] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:19:26.407797 1303 tablet_replica.cc:330] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: stopping tablet replica
I20260810 04:19:26.407790 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 810
I20260810 04:19:26.407910 1303 raft_consensus.cc:2229] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:19:26.407979 987 ts_tablet_manager.cc:1905] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:19:26.408012 1303 raft_consensus.cc:2258] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:19:26.408603 1303 ts_tablet_manager.cc:1905] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:19:26.410624 1302 tablet_replica.cc:330] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: stopping tablet replica
I20260810 04:19:26.410761 1302 raft_consensus.cc:2229] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:19:26.410924 1302 raft_consensus.cc:2258] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:19:26.411132 1303 ts_tablet_manager.cc:1918] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:19:26.411242 1303 log.cc:1196] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:26.411586 1302 ts_tablet_manager.cc:1905] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:19:26.411633 1303 ts_tablet_manager.cc:1939] T aadf3fefffd6421f8a15e8f0141828ac P 61ea357ec632432e89661aae77d70395: Deleting consensus metadata
I20260810 04:19:26.412154 755 catalog_manager.cc:4841] TS 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): tablet aadf3fefffd6421f8a15e8f0141828ac (table test-table [id=b61eb28452c84588a8c3e0eac05b566b]) successfully deleted
I20260810 04:19:26.414072 1302 ts_tablet_manager.cc:1918] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:19:26.414162 1302 log.cc:1196] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:26.414463 1302 ts_tablet_manager.cc:1939] T aadf3fefffd6421f8a15e8f0141828ac P 1df1c60e8f844f8584f63177d28f5a1f: Deleting consensus metadata
I20260810 04:19:26.415041 753 catalog_manager.cc:4841] TS 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): tablet aadf3fefffd6421f8a15e8f0141828ac (table test-table [id=b61eb28452c84588a8c3e0eac05b566b]) successfully deleted
W20260810 04:19:26.418123 753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:38293 (error 108)
I20260810 04:19:26.418437 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 983
I20260810 04:19:26.419488 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:26.418526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:38293 (local address 127.0.0.1:59962)
0810 04:19:26.419393 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:26.419444 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":841}
W20260810 04:19:26.419948 753 catalog_manager.cc:4584] TS e3dc229a78bd4717af6c964f9fa94300 (127.15.216.65:38293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aadf3fefffd6421f8a15e8f0141828ac: Network error: Client connection negotiation failed: client connection to 127.15.216.65:38293: connect: Connection refused (error 111)
I20260810 04:19:26.427660 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 1130
I20260810 04:19:26.436858 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 734
I20260810 04:19:46.446532 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41741
--webserver_interface=127.15.216.126
--webserver_port=38899
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:46.532055 1305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:46.532401 1305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:46.532445 1305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:46.533716 1305 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 04:19:46.533800 1305 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 04:19:46.533821 1305 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 04:19:46.533844 1305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:19:46.533866 1305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:46.533890 1305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:46.533921 1305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:19:46.533950 1305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:19:46.535610 1305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=38899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:46.535955 1305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:46.536216 1305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:46.542294 1313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:46.542294 1312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:46.542366 1315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:46.542501 1305 server_base.cc:1034] running on GCE node
I20260810 04:19:46.542788 1305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:46.543102 1305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:46.544301 1305 hybrid_clock.cc:648] HybridClock initialized: now 1786335586544282 us; error 27 us; skew 500 ppm
Aug 10 04:19:46 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335586, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:46.553126 1305 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:19:46.553759 1305 webserver.cc:460] Webserver started at http://127.15.216.126:38899/ using document root <none> and password file <none>
I20260810 04:19:46.554050 1305 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:46.554131 1305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:46.555797 1305 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:19:46.556677 1322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:46.556944 1305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:46.557076 1305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e5ef2b69afaa4404b4214f25d8b06087"
format_stamp: "Formatted at 2026-08-10 04:19:04 on dist-test-slave-g350"
server_key: "800d42379678694ba01af291e7c5a1ef"
server_key_iv: "fbdb6846ada38ba5fe0738c21dd5dc59"
server_key_version: "encryptionkey@0"
I20260810 04:19:46.557493 1305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:46.570181 1305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:46.572803 1305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:46.572986 1305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:46.577761 1305 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41741
I20260810 04:19:46.577816 1378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41741 every 8 connection(s)
I20260810 04:19:46.578243 1305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:19:46.579916 1379 sys_catalog.cc:263] Verifying existing consensus state
I20260810 04:19:46.580664 1379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Bootstrap starting.
I20260810 04:19:46.583540 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 1305
I20260810 04:19:46.583914 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:38293
--local_ip_for_outbound_sockets=127.15.216.65
--tserver_master_addrs=127.15.216.126:41741
--webserver_port=38161
--webserver_interface=127.15.216.65
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:19:46.591163 1379 log.cc:824] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:46.593528 1379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:19:46.593900 1379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Bootstrap complete.
I20260810 04:19:46.596308 1379 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:46.596542 1379 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5ef2b69afaa4404b4214f25d8b06087, State: Initialized, Role: FOLLOWER
I20260810 04:19:46.596706 1379 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:46.596791 1379 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:19:46.596824 1379 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:19:46.596889 1379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 2 FOLLOWER]: Advancing to term 3
I20260810 04:19:46.598035 1379 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:46.598168 1379 leader_election.cc:302] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5ef2b69afaa4404b4214f25d8b06087; no voters:
I20260810 04:19:46.598374 1379 leader_election.cc:288] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 04:19:46.598459 1384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 04:19:46.598825 1379 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:19:46.598836 1384 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [term 3 LEADER]: Becoming Leader. State: Replica: e5ef2b69afaa4404b4214f25d8b06087, State: Running, Role: LEADER
I20260810 04:19:46.599062 1384 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } }
I20260810 04:19:46.599347 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5ef2b69afaa4404b4214f25d8b06087. Latest consensus state: current_term: 3 leader_uuid: "e5ef2b69afaa4404b4214f25d8b06087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } } }
I20260810 04:19:46.599418 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: This master's current role is: LEADER
I20260810 04:19:46.599352 1385 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e5ef2b69afaa4404b4214f25d8b06087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5ef2b69afaa4404b4214f25d8b06087" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41741 } } }
I20260810 04:19:46.599462 1385 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087 [sys.catalog]: This master's current role is: LEADER
I20260810 04:19:46.599752 1390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:19:46.600793 1390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:19:46.600997 1390 catalog_manager.cc:1259] Loaded cluster ID: dec24dcbad3242699f8e9cab514a1d7c
I20260810 04:19:46.601187 1390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:19:46.602568 1390 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:19:46.602756 1390 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Loaded TSK: 0
I20260810 04:19:46.602800 1390 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5ef2b69afaa4404b4214f25d8b06087: Loaded TSK: 1
F20260810 04:19:46.603032 1390 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f852c5999ad google::LogMessage::Fail() at ??:0
@ 0x7f852c59c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f852c5994de google::LogMessage::Flush() at ??:0
@ 0x7f852c59aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f852ed06104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f852ed03d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f85344f71cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f85344f748e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f85344f7594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f85344bdc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f85344c7ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f852e4670d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f852e45c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f852f2c06db start_thread at ??:0
@ 0x7f852b09761f clone at ??:0
W20260810 04:19:46.707878 1382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:46.708177 1382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:46.708218 1382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:46.710057 1382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:46.710131 1382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:46.710180 1382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:19:46.712428 1382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:38293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=38161
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:46.712711 1382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:46.713001 1382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:46.718773 1424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:46.719022 1382 server_base.cc:1034] running on GCE node
W20260810 04:19:46.718878 1427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:46.718778 1425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:46.719376 1382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:46.719653 1382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:46.720841 1382 hybrid_clock.cc:648] HybridClock initialized: now 1786335586720812 us; error 43 us; skew 500 ppm
Aug 10 04:19:46 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335586, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:46.724016 1382 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:19:46.724467 1382 webserver.cc:460] Webserver started at http://127.15.216.65:38161/ using document root <none> and password file <none>
I20260810 04:19:46.724756 1382 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:46.724839 1382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:46.726346 1382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:19:46.727185 1435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:46.727373 1382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:46.727485 1382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "e3dc229a78bd4717af6c964f9fa94300"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "8cb5e653b4f24dd3e873698994f5b82c"
server_key_iv: "0ee9e437ab93174de9c4d6affa745cce"
server_key_version: "encryptionkey@0"
I20260810 04:19:46.727849 1382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:46.751269 1382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:46.754103 1382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:46.754285 1382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:46.754591 1382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:46.755139 1443 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 04:19:46.756697 1382 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 04:19:46.756752 1382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 04:19:46.756779 1382 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 04:19:46.757469 1382 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 04:19:46.757514 1382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 04:19:46.757562 1443 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Bootstrap starting.
I20260810 04:19:46.759948 1443 log.cc:824] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Log is configured to *not* fsync() on all Append() calls
I20260810 04:19:46.763550 1443 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 04:19:46.763919 1443 tablet_bootstrap.cc:492] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Bootstrap complete.
I20260810 04:19:46.764101 1443 ts_tablet_manager.cc:1397] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 04:19:46.766108 1382 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:38293
I20260810 04:19:46.766151 1562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:38293 every 8 connection(s)
I20260810 04:19:46.766469 1382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:19:46.766366 1443 raft_consensus.cc:348] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:46.766705 1443 raft_consensus.cc:729] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dc229a78bd4717af6c964f9fa94300, State: Initialized, Role: FOLLOWER
I20260810 04:19:46.767045 1443 consensus_queue.cc:260] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:46.767362 1443 ts_tablet_manager.cc:1428] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:19:46.771350 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 1382
I20260810 04:19:46.771695 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:41665
--local_ip_for_outbound_sockets=127.15.216.66
--tserver_master_addrs=127.15.216.126:41741
--webserver_port=46471
--webserver_interface=127.15.216.66
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:46.856791 1567 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:19:46.767315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:57869)
0810 04:19:46.767489 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:46.767534 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:46.767689 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:46.856569 (+ 88880us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
W20260810 04:19:46.857126 1563 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:41741 (0 consecutive failures): Network error: Failed to ping master at 127.15.216.126:41741: Client connection negotiation failed: client connection to 127.15.216.126:41741: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 04:19:46.857559 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:46.857413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:43845)
0810 04:19:46.857476 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:46.857527 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:19:46.858057 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:46.857906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:58909)
0810 04:19:46.857981 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:46.858020 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 04:19:46.858223 1563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 04:19:46.878731 1568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:46.878988 1568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:46.879033 1568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:46.880241 1568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:46.880314 1568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:46.880371 1568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:19:46.881968 1568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:41665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=46471
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:46.882246 1568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:46.882498 1568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:46.888118 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:46.888113 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:46.888145 1574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:46.888198 1568 server_base.cc:1034] running on GCE node
I20260810 04:19:46.888691 1568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:46.888949 1568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:46.890127 1568 hybrid_clock.cc:648] HybridClock initialized: now 1786335586890119 us; error 36 us; skew 500 ppm
Aug 10 04:19:46 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335586, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:46.892828 1568 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:19:46.893270 1568 webserver.cc:460] Webserver started at http://127.15.216.66:46471/ using document root <none> and password file <none>
I20260810 04:19:46.893536 1568 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:46.893626 1568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:46.895010 1568 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:19:46.895644 1585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:46.895828 1568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:19:46.895947 1568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "61ea357ec632432e89661aae77d70395"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "e8d28ced4137e160b2f0947023f3a6be"
server_key_iv: "4787a3996b96de2afa7fbb3a3184276e"
server_key_version: "encryptionkey@0"
I20260810 04:19:46.896303 1568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:46.916095 1568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:46.918675 1568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:46.918854 1568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:46.919108 1568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:46.919438 1568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:19:46.919493 1568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:46.919559 1568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:19:46.919606 1568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:46.925797 1568 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:41665
I20260810 04:19:46.925843 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:41665 every 8 connection(s)
I20260810 04:19:46.926263 1568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:19:46.927369 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:46.926967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:48753)
0810 04:19:46.927149 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:46.927222 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
W20260810 04:19:46.927793 1710 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:41741 (0 consecutive failures): Network error: Failed to ping master at 127.15.216.126:41741: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
I20260810 04:19:46.928071 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:46.927981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:52433)
0810 04:19:46.928026 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:46.928052 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 04:19:46.928332 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:46.928276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:51281)
0810 04:19:46.928299 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:46.928319 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260810 04:19:46.928444 1710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:19:46.929852 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 1568
I20260810 04:19:46.930138 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:34923
--local_ip_for_outbound_sockets=127.15.216.67
--tserver_master_addrs=127.15.216.126:41741
--webserver_port=40909
--webserver_interface=127.15.216.67
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:19:47.016077 1713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:19:47.016294 1713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:19:47.016319 1713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:19:47.017822 1713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:19:47.017876 1713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:19:47.017905 1713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:19:47.019423 1713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:34923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=40909
--tserver_master_addrs=127.15.216.126:41741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:19:47.019680 1713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:19:47.019943 1713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:19:47.025429 1718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:47.025421 1720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:19:47.025627 1722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:19:47.025451 1713 server_base.cc:1034] running on GCE node
I20260810 04:19:47.025960 1713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:19:47.026239 1713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:19:47.027415 1713 hybrid_clock.cc:648] HybridClock initialized: now 1786335587027404 us; error 48 us; skew 500 ppm
Aug 10 04:19:47 dist-test-slave-g350 krb5kdc[32609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335587, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:19:47.030210 1713 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:19:47.030750 1713 webserver.cc:460] Webserver started at http://127.15.216.67:40909/ using document root <none> and password file <none>
I20260810 04:19:47.031021 1713 fs_manager.cc:359] Metadata directory not provided
I20260810 04:19:47.031085 1713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:19:47.032428 1713 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:19:47.033105 1729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:47.033288 1713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:19:47.033401 1713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "1df1c60e8f844f8584f63177d28f5a1f"
format_stamp: "Formatted at 2026-08-10 04:19:05 on dist-test-slave-g350"
server_key: "c2e77c52e9ef3ca1e38af7aedccda69d"
server_key_iv: "03269c382035b7bc756359e55c757776"
server_key_version: "encryptionkey@0"
I20260810 04:19:47.033838 1713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:19:47.041242 1713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:19:47.043605 1713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:19:47.043720 1713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:19:47.043924 1713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:19:47.044243 1713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:19:47.044277 1713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:47.044301 1713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:19:47.044337 1713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:19:47.050577 1713 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:34923
I20260810 04:19:47.050685 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:34923 every 8 connection(s)
I20260810 04:19:47.051023 1713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:19:47.052126 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.051752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:57635)
0810 04:19:47.051905 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:47.051984 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
W20260810 04:19:47.052565 1856 heartbeater.cc:644] Failed to heartbeat to 127.15.216.126:41741 (0 consecutive failures): Network error: Failed to ping master at 127.15.216.126:41741: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
I20260810 04:19:47.052877 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.052765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:41541)
0810 04:19:47.052829 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:47.052858 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 04:19:47.053222 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.053167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:54207)
0810 04:19:47.053190 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:47.053209 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260810 04:19:47.053342 1856 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 04:19:47.056180 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 1713
I20260810 04:19:47.858960 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.858670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:42533)
0810 04:19:47.858843 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:47.858887 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260810 04:19:47.929185 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.928937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:58027)
0810 04:19:47.929080 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:47.929125 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260810 04:19:47.994516 1863 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:47.994658 1863 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:47.994968 1863 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
Aug 10 04:19:47 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335586, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.66@KRBTEST.COM
Aug 10 04:19:47 dist-test-slave-g350 krb5kdc[32609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335586, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.67@KRBTEST.COM
I20260810 04:19:48.009416 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.995355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:47543 (local address 127.15.216.67:34923)
0810 04:19:47.995530 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:47.995543 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:47.995568 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:19:47.996194 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:47.996198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:47.996264 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:47.996428 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:47.997921 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:47.999290 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:48.001074 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:48.001742 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:48.001867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:48.005259 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:48.005262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:48.005263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:48.005273 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:48.007261 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:48.007889 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:48.007893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:48.007893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:48.008029 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:48.008416 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:48.008418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:48.008418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:48.008704 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:48.008770 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:48.009135 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 04:19:48.009281 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260810 04:19:48.009408 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.995211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41665 (local address 127.15.216.65:46599)
0810 04:19:47.995375 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:47.995386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:47.995448 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:47.996522 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:47.996526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:47.997142 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:47.997538 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:47.997551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:47.999374 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:47.999375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:48.000951 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:48.000955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:48.001914 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:48.001915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:48.002088 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:48.002808 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:48.002812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:48.004774 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:48.007700 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:48.007701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:48.007701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:48.007959 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:48.008286 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:48.008287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:48.008287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:48.008402 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:48.008874 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:48.008875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:48.008954 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 04:19:48.009073 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 04:19:48.009291 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":118,"thread_start_us":87,"threads_started":1}
I20260810 04:19:48.009501 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.995436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:46599 (local address 127.15.216.66:41665)
0810 04:19:47.995587 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 04:19:47.995595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:19:47.995626 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:19:47.996200 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:19:47.996203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:19:47.996260 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:19:47.996428 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:19:47.997730 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:47.999150 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:48.001074 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:19:48.001742 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:19:48.001883 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:48.005224 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:19:48.005234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:19:48.005235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:19:48.005250 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:19:48.007599 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:48.008049 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:48.008052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:48.008053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:48.008208 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:19:48.008477 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:19:48.008477 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:19:48.008478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:19:48.008726 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:19:48.008875 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 04:19:48.009140 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 04:19:48.009365 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260810 04:19:48.010296 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:47.995209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34923 (local address 127.15.216.65:47543)
0810 04:19:47.995290 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:47.995306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:19:47.995435 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:19:47.996522 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:19:47.996526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:19:47.997142 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:19:47.997807 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:47.997810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:47.999394 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:47.999395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:48.000951 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:19:48.000955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:19:48.001913 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:19:48.001915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:19:48.002088 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:19:48.002772 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:19:48.002785 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:19:48.005156 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:19:48.007391 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:48.007402 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:48.007405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:48.007779 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:48.008111 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:19:48.008112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:19:48.008112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:19:48.008330 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:19:48.008786 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:19:48.008794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:19:48.008967 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 04:19:48.009073 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 04:19:48.009395 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":234}
W20260810 04:19:48.010759 1439 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
W20260810 04:19:48.011057 1437 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:48.011119 1437 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3dc229a78bd4717af6c964f9fa94300; no voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395
I20260810 04:19:48.011260 1863 raft_consensus.cc:2726] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:19:48.054054 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:48.053750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:54017)
0810 04:19:48.053956 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:48.054006 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":121,"threads_started":1}
I20260810 04:19:48.859913 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:48.859655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:45227)
0810 04:19:48.859815 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:48.859863 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260810 04:19:48.929971 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:48.929754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:51539)
0810 04:19:48.929892 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:48.929925 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260810 04:19:49.054996 1871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:49.054693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:43793)
0810 04:19:49.054879 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:49.054930 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
I20260810 04:19:49.546623 1872 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:49.546761 1872 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:49.547019 1872 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
W20260810 04:19:49.547546 1437 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
W20260810 04:19:49.547663 1439 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:49.547727 1439 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3dc229a78bd4717af6c964f9fa94300; no voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395
I20260810 04:19:49.547792 1872 raft_consensus.cc:2726] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:19:49.861148 1873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:49.860777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:54139)
0810 04:19:49.860975 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:49.861050 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
I20260810 04:19:49.931102 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:49.930653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:57627)
0810 04:19:49.930891 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:49.930994 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":138,"threads_started":1}
I20260810 04:19:50.056685 1876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:50.055943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:42673)
0810 04:19:50.056273 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:50.056526 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":95,"threads_started":1}
I20260810 04:19:50.862262 1877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:50.862003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:43635)
0810 04:19:50.862160 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:50.862213 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260810 04:19:50.932102 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:50.931876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:58217)
0810 04:19:50.932008 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:50.932053 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260810 04:19:51.057742 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:51.057395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:34105)
0810 04:19:51.057583 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:51.057689 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
I20260810 04:19:51.863154 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:51.862899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:39623)
0810 04:19:51.863057 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:51.863104 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260810 04:19:51.932886 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:51.932636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:41355)
0810 04:19:51.932789 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:51.932837 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260810 04:19:51.942662 1883 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:51.942777 1883 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:51.942965 1883 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
W20260810 04:19:51.943346 1439 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
W20260810 04:19:51.943416 1437 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:51.943445 1439 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3dc229a78bd4717af6c964f9fa94300; no voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395
I20260810 04:19:51.943627 1883 raft_consensus.cc:2726] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:19:52.058562 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:52.058290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:47459)
0810 04:19:52.058478 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:52.058514 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":118,"threads_started":1}
I20260810 04:19:52.864130 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:52.863843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:60243)
0810 04:19:52.864016 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:52.864079 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260810 04:19:52.933717 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:52.933433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:38021)
0810 04:19:52.933574 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:52.933669 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260810 04:19:53.059291 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:53.059059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:45683)
0810 04:19:53.059202 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:53.059241 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260810 04:19:53.573863 1889 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:53.573971 1889 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:53.574150 1889 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
W20260810 04:19:53.574641 1437 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
W20260810 04:19:53.574724 1439 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:53.574759 1439 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3dc229a78bd4717af6c964f9fa94300; no voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395
I20260810 04:19:53.574846 1889 raft_consensus.cc:2726] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:19:53.864986 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:53.864752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:36487)
0810 04:19:53.864890 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:53.864933 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260810 04:19:53.934504 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:53.934187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:56437)
0810 04:19:53.934384 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:53.934433 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260810 04:19:54.060103 1892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:54.059821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:58177)
0810 04:19:54.060007 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:54.060051 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
I20260810 04:19:54.866014 1893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:54.865714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:43655)
0810 04:19:54.865902 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:54.865951 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20260810 04:19:54.935360 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:54.935066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:34727)
0810 04:19:54.935246 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:54.935309 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260810 04:19:55.061002 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:55.060734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:40695)
0810 04:19:55.060900 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:55.060950 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260810 04:19:55.866883 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:55.866606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:51135)
0810 04:19:55.866769 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:55.866817 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260810 04:19:55.936182 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:55.935950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:56147)
0810 04:19:55.936084 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:55.936130 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260810 04:19:56.061859 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:56.061564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:34331)
0810 04:19:56.061760 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:56.061807 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":126,"threads_started":1}
I20260810 04:19:56.643743 1899 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:19:56.643872 1899 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:19:56.644122 1899 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
W20260810 04:19:56.644613 1437 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
W20260810 04:19:56.644675 1439 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:19:56.644703 1437 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3dc229a78bd4717af6c964f9fa94300; no voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395
I20260810 04:19:56.644793 1899 raft_consensus.cc:2726] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:19:56.867831 1900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:56.867552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:48869)
0810 04:19:56.867731 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:56.867779 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260810 04:19:56.937103 1901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:56.936832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:47389)
0810 04:19:56.937002 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:56.937051 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260810 04:19:57.062757 1902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:57.062461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:37653)
0810 04:19:57.062636 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:57.062702 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260810 04:19:57.868777 1903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:57.868492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:34981)
0810 04:19:57.868675 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:57.868723 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260810 04:19:57.938000 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:57.937764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:55507)
0810 04:19:57.937900 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:57.937949 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260810 04:19:58.063570 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:58.063336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:38829)
0810 04:19:58.063476 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:58.063519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260810 04:19:58.869707 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:58.869440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:41983)
0810 04:19:58.869606 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:58.869655 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260810 04:19:58.938781 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:58.938529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:38607)
0810 04:19:58.938677 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:58.938732 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260810 04:19:59.064402 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:59.064134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:46779)
0810 04:19:59.064295 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:59.064344 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260810 04:19:59.870610 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:59.870350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:35489)
0810 04:19:59.870513 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:59.870560 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260810 04:19:59.939602 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:19:59.939305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:37309)
0810 04:19:59.939468 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 04:19:59.939519 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260810 04:20:00.065302 1913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:00.065025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:36727)
0810 04:20:00.065200 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:00.065243 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260810 04:20:00.871546 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:00.871240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:56579)
0810 04:20:00.871443 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:00.871495 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":145,"threads_started":1}
I20260810 04:20:00.940428 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:00.940135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:34677)
0810 04:20:00.940317 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:00.940374 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
I20260810 04:20:01.066233 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:01.065978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:51613)
0810 04:20:01.066137 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:01.066179 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260810 04:20:01.499698 1917 raft_consensus.cc:482] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:01.499807 1917 raft_consensus.cc:504] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ea357ec632432e89661aae77d70395" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41665 } } peers { permanent_uuid: "1df1c60e8f844f8584f63177d28f5a1f" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34923 } } peers { permanent_uuid: "e3dc229a78bd4717af6c964f9fa94300" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 38293 } }
I20260810 04:20:01.500003 1917 leader_election.cc:288] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665), 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923)
W20260810 04:20:01.500473 1437 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1df1c60e8f844f8584f63177d28f5a1f (127.15.216.67:34923): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
W20260810 04:20:01.500562 1439 leader_election.cc:341] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61ea357ec632432e89661aae77d70395 (127.15.216.66:41665): Not found: Tablet not found: aadf3fefffd6421f8a15e8f0141828ac
I20260810 04:20:01.500651 1439 leader_election.cc:302] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3dc229a78bd4717af6c964f9fa94300; no voters: 1df1c60e8f844f8584f63177d28f5a1f, 61ea357ec632432e89661aae77d70395
I20260810 04:20:01.500751 1917 raft_consensus.cc:2726] T aadf3fefffd6421f8a15e8f0141828ac P e3dc229a78bd4717af6c964f9fa94300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 04:20:01.872514 1918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:01.872247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.65:44319)
0810 04:20:01.872425 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:01.872466 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260810 04:20:01.941452 1919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:01.941101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.66:42967)
0810 04:20:01.941333 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:01.941382 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
I20260810 04:20:02.061013 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 1382
I20260810 04:20:02.067189 1920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:02.066947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41741 (local address 127.15.216.67:35199)
0810 04:20:02.067079 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:02.067120 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.126:41741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260810 04:20:02.073074 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 1568
I20260810 04:20:02.085173 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 1713
2026-08-10T04:20:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58598 ms)
[----------] 30 tests from SecurityITest (239653 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:02 dist-test-slave-g350 krb5kdc[1923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:02 dist-test-slave-g350 krb5kdc[1923](info): set up 2 sockets
Aug 10 04:20:02 dist-test-slave-g350 krb5kdc[1923](info): commencing operation
krb5kdc: starting...
W20260810 04:20:03.185230 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:03.316355 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41247
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:03.404886 1938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:03.405115 1938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:03.405139 1938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:03.406440 1938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:03.406490 1938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:03.406505 1938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:03.406518 1938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:03.406531 1938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:03.408085 1938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:03.408358 1938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:03.408618 1938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:03.413776 1946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:03.413776 1943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:03.413777 1944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:03.414206 1938 server_base.cc:1034] running on GCE node
I20260810 04:20:03.414481 1938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:03.414788 1938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:03.415962 1938 hybrid_clock.cc:648] HybridClock initialized: now 1786335603415960 us; error 42 us; skew 500 ppm
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:03.418617 1938 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:03.419061 1938 webserver.cc:460] Webserver started at http://127.15.216.126:35213/ using document root <none> and password file <none>
I20260810 04:20:03.419317 1938 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:03.419394 1938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:03.419546 1938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:03.420619 1938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "e6b4756c9acd415491ce78f178661fcd"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "e508d812745b5719956ed4cb6fec125f"
server_key_iv: "d1b73f282188ba7e055c8efa646c4996"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.420994 1938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "e6b4756c9acd415491ce78f178661fcd"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "e508d812745b5719956ed4cb6fec125f"
server_key_iv: "d1b73f282188ba7e055c8efa646c4996"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.422500 1938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:03.423378 1953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.423601 1938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:03.423700 1938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e6b4756c9acd415491ce78f178661fcd"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "e508d812745b5719956ed4cb6fec125f"
server_key_iv: "d1b73f282188ba7e055c8efa646c4996"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.423785 1938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:03.449378 1938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:03.452148 1938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:03.452390 1938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:03.456823 1938 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41247
I20260810 04:20:03.456936 2012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41247 every 8 connection(s)
I20260810 04:20:03.457314 1938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:03.458290 2013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:03.460623 2013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd: Bootstrap starting.
I20260810 04:20:03.461230 2013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:03.461525 2013 log.cc:824] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:03.462265 2013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd: No bootstrap required, opened a new log
I20260810 04:20:03.463903 2013 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b4756c9acd415491ce78f178661fcd" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41247 } }
I20260810 04:20:03.464056 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 1938
I20260810 04:20:03.464074 2013 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:03.464116 2013 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b4756c9acd415491ce78f178661fcd, State: Initialized, Role: FOLLOWER
I20260810 04:20:03.464193 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:03.464316 2013 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b4756c9acd415491ce78f178661fcd" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41247 } }
I20260810 04:20:03.464430 2013 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:03.464421 16225 external_mini_cluster.cc:1428] Setting key cf22f2385e717d33bf44fee145c63875
I20260810 04:20:03.464478 2013 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:03.464529 2013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:03.465680 2013 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b4756c9acd415491ce78f178661fcd" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41247 } }
I20260810 04:20:03.465806 2013 leader_election.cc:302] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6b4756c9acd415491ce78f178661fcd; no voters:
I20260810 04:20:03.465988 2013 leader_election.cc:288] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:03.466048 2019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:03.466238 2019 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [term 1 LEADER]: Becoming Leader. State: Replica: e6b4756c9acd415491ce78f178661fcd, State: Running, Role: LEADER
I20260810 04:20:03.466262 2013 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:03.466347 2019 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b4756c9acd415491ce78f178661fcd" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41247 } }
I20260810 04:20:03.467056 2021 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6b4756c9acd415491ce78f178661fcd. Latest consensus state: current_term: 1 leader_uuid: "e6b4756c9acd415491ce78f178661fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b4756c9acd415491ce78f178661fcd" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41247 } } }
I20260810 04:20:03.467165 2021 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:03.467537 2028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:03.467540 2020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6b4756c9acd415491ce78f178661fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b4756c9acd415491ce78f178661fcd" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41247 } } }
I20260810 04:20:03.467990 2020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:03.468185 2028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:03.469787 2028 catalog_manager.cc:1347] Generated new cluster ID: 90de7cb5942b47e798b04f7cf9f163cc
I20260810 04:20:03.469847 2028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:03.476655 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:03.464980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.15.216.126:41247)
0810 04:20:03.465315 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:03.465323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:03.465350 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:20:03.465443 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:03.465449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:03.465491 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:03.465659 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:03.466549 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.468085 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.469142 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.469741 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.469853 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:03.472840 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:03.472845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:03.472846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:03.472866 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:03.474884 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:03.475333 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:03.475336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:03.475337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:03.475444 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:03.475686 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:03.475688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:03.475688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:03.475984 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:03.476161 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:03.476298 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 04:20:03.476472 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":185,"threads_started":1}
I20260810 04:20:03.486195 2028 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:03.486994 2028 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:03.494128 2028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd: Generated new TSK 0
I20260810 04:20:03.494354 2028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:20:03.525257 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41247
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:03.615181 2040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:03.615391 2040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:03.615415 2040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:03.616596 2040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:03.616642 2040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:03.616668 2040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:20:03.618769 2040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:03.619046 2040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:03.619300 2040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:03.624766 2048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:03.624780 2046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:03.624773 2045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:03.624948 2040 server_base.cc:1034] running on GCE node
I20260810 04:20:03.625204 2040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:03.625468 2040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:03.626648 2040 hybrid_clock.cc:648] HybridClock initialized: now 1786335603626628 us; error 31 us; skew 500 ppm
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:03.629185 2040 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:20:03.629691 2040 webserver.cc:460] Webserver started at http://127.15.216.65:33063/ using document root <none> and password file <none>
I20260810 04:20:03.629956 2040 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:03.630003 2040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:03.630154 2040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:03.631213 2040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "3ce8e90bc1de60f25022f1b88641aa65"
server_key_iv: "be4f52e4b00df5797075f4df71cb9e01"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.631644 2040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "3ce8e90bc1de60f25022f1b88641aa65"
server_key_iv: "be4f52e4b00df5797075f4df71cb9e01"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.633015 2040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:03.634085 2055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.634297 2040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:03.634379 2040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "3ce8e90bc1de60f25022f1b88641aa65"
server_key_iv: "be4f52e4b00df5797075f4df71cb9e01"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.634428 2040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:03.643908 2040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:03.646191 2040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:03.646308 2040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:03.646541 2040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:03.646884 2040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:03.646943 2040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.646981 2040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:03.647024 2040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.653138 2040 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:42739
I20260810 04:20:03.653220 2178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:42739 every 8 connection(s)
I20260810 04:20:03.653573 2040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:03.661926 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 2040
I20260810 04:20:03.662058 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:20:03.662364 16225 external_mini_cluster.cc:1428] Setting key 16c2c321ebf44ad87a08db92ac6b804f
I20260810 04:20:03.664342 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:03.654448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:39335 (local address 127.15.216.126:41247)
0810 04:20:03.654522 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:03.654524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:03.654725 (+ 201us) server_negotiation.cc:407] Connection header received
0810 04:20:03.655203 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:03.655204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:03.655218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:03.655283 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:03.656379 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.656908 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.658085 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.658315 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.658384 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:03.660818 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:03.660820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:03.660820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:03.660826 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:03.662760 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:03.663225 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:03.663226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:03.663227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:03.663299 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:03.663589 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:03.663590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:03.663590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:03.663726 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:03.663818 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:03.664139 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 04:20:03.664233 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:20:03.664577 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:03.654406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41247 (local address 127.15.216.65:39335)
0810 04:20:03.654573 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:03.654616 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:03.655118 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:03.655433 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:03.655437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:03.656035 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:03.656275 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:03.656280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.657030 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.657031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:03.657965 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:03.657968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.658430 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.658431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:03.658550 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:03.659093 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:03.659102 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:03.660676 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:03.662864 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:03.662873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:03.662875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:03.663131 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:03.663381 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:03.663383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:03.663383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:03.663508 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:03.663810 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:03.663818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:03.663974 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 04:20:03.664112 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 04:20:03.664310 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260810 04:20:03.665300 2179 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41247
I20260810 04:20:03.665422 2179 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:03.665695 2179 heartbeater.cc:511] Master 127.15.216.126:41247 requested a full tablet report, sending...
I20260810 04:20:03.666371 1973 ts_manager.cc:194] Registered new tserver with Master: cf09d3202ce8484d9b8a8b4eeeb5ef9a (127.15.216.65:42739)
I20260810 04:20:03.667181 1973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:39335
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:20:03.694237 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41247
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:03.780206 2184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:03.780453 2184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:03.780500 2184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:03.781737 2184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:03.781805 2184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:03.781850 2184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:20:03.783591 2184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:03.783900 2184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:03.784157 2184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:03.789564 2189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:03.789673 2192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:03.789870 2184 server_base.cc:1034] running on GCE node
W20260810 04:20:03.789583 2190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:03.790213 2184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:03.790475 2184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:03.791663 2184 hybrid_clock.cc:648] HybridClock initialized: now 1786335603791648 us; error 59 us; skew 500 ppm
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:03.794242 2184 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:20:03.794693 2184 webserver.cc:460] Webserver started at http://127.15.216.66:38237/ using document root <none> and password file <none>
I20260810 04:20:03.794953 2184 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:03.795022 2184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:03.795167 2184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:03.796236 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "3e8e951d2489435fb7422c20f4b41d9c"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "04a1d81adb54aa53c1aa98a8125e7e17"
server_key_iv: "8c3dfc69afc06a91a25f3368d798b33a"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.796605 2184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "3e8e951d2489435fb7422c20f4b41d9c"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "04a1d81adb54aa53c1aa98a8125e7e17"
server_key_iv: "8c3dfc69afc06a91a25f3368d798b33a"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.798027 2184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:03.798822 2201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.799002 2184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:03.799111 2184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "3e8e951d2489435fb7422c20f4b41d9c"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "04a1d81adb54aa53c1aa98a8125e7e17"
server_key_iv: "8c3dfc69afc06a91a25f3368d798b33a"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.799186 2184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:03.811257 2184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:03.813270 2184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:03.813446 2184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:03.813742 2184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:03.814061 2184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:03.814117 2184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.814162 2184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:03.814188 2184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.820475 2184 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:38179
I20260810 04:20:03.820573 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:38179 every 8 connection(s)
I20260810 04:20:03.820878 2184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:03.830173 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:03.821641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:36041 (local address 127.15.216.126:41247)
0810 04:20:03.821688 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:03.821690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:03.821930 (+ 240us) server_negotiation.cc:407] Connection header received
0810 04:20:03.822409 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:03.822411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:03.822422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:03.822484 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:03.823479 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.823984 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.825005 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.825208 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.825299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:03.827625 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:03.827627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:03.827627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:03.827634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:03.828866 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:03.829250 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:03.829251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:03.829251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:03.829302 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:03.829529 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:03.829530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:03.829530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:03.829701 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:03.829756 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:03.830038 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 04:20:03.830103 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:20:03.830366 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:03.821561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41247 (local address 127.15.216.66:36041)
0810 04:20:03.821773 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:03.821817 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:03.822316 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:03.822535 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:03.822538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:03.823113 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:03.823357 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:03.823364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.824089 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.824090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:03.824905 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:03.824907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:03.825313 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:03.825314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:03.825445 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:03.826032 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:03.826041 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:03.827510 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:03.828942 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:03.828949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:03.828951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:03.829183 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:03.829366 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:03.829367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:03.829367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:03.829472 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:03.829761 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:03.829766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:03.829886 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 04:20:03.830014 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 04:20:03.830144 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":144,"threads_started":1}
I20260810 04:20:03.830464 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 2184
I20260810 04:20:03.830550 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:20:03.830821 16225 external_mini_cluster.cc:1428] Setting key 2e8bf230f17e8079eb80b2823874543d
I20260810 04:20:03.830992 2326 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41247
I20260810 04:20:03.831096 2326 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:03.831292 2326 heartbeater.cc:511] Master 127.15.216.126:41247 requested a full tablet report, sending...
I20260810 04:20:03.831754 1973 ts_manager.cc:194] Registered new tserver with Master: 3e8e951d2489435fb7422c20f4b41d9c (127.15.216.66:38179)
I20260810 04:20:03.832147 1973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:36041
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:20:03.862334 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41247
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:03.952183 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:03.952469 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:03.952504 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:03.954182 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:03.954268 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:03.954321 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:20:03.956606 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:03.956904 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:03.957165 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:03.962778 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:03.962806 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:03.962815 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:03.963002 2333 server_base.cc:1034] running on GCE node
I20260810 04:20:03.963204 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:03.963485 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:03.964715 2333 hybrid_clock.cc:648] HybridClock initialized: now 1786335603964698 us; error 25 us; skew 500 ppm
Aug 10 04:20:03 dist-test-slave-g350 krb5kdc[1923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:03.967927 2333 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:20:03.968457 2333 webserver.cc:460] Webserver started at http://127.15.216.67:45643/ using document root <none> and password file <none>
I20260810 04:20:03.968760 2333 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:03.968850 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:03.969034 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:03.970378 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "7212eda7ff3a4f278b7afb67fb9a30ec"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "abd928e43c64d8877e5cbb0ad23f00f2"
server_key_iv: "b62cfa0080887661247ae25e29daf862"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.970817 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "7212eda7ff3a4f278b7afb67fb9a30ec"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "abd928e43c64d8877e5cbb0ad23f00f2"
server_key_iv: "b62cfa0080887661247ae25e29daf862"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.972422 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:20:03.973387 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.973690 2333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:03.973835 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "7212eda7ff3a4f278b7afb67fb9a30ec"
format_stamp: "Formatted at 2026-08-10 04:20:03 on dist-test-slave-g350"
server_key: "abd928e43c64d8877e5cbb0ad23f00f2"
server_key_iv: "b62cfa0080887661247ae25e29daf862"
server_key_version: "encryptionkey@0"
I20260810 04:20:03.973960 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:03.989974 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:03.992317 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:03.992444 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:03.992666 2333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:03.992981 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:03.993016 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:03.993045 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:03.993060 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.000067 2333 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:40079
I20260810 04:20:04.000125 2474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:40079 every 8 connection(s)
I20260810 04:20:04.000488 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:20:04 dist-test-slave-g350 krb5kdc[1923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335603, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:04.008783 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 2333
I20260810 04:20:04.008924 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:20:04.009124 16225 external_mini_cluster.cc:1428] Setting key 81f302ce164ef2ad54769120f8152ad8
I20260810 04:20:04.011101 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.001279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:42827 (local address 127.15.216.126:41247)
0810 04:20:04.001349 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.001351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.001580 (+ 229us) server_negotiation.cc:407] Connection header received
0810 04:20:04.002312 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.002313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.002326 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.002398 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:04.003526 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.004021 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.005070 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.005254 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.005317 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.007707 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:04.007708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:04.007708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:04.007715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:04.009448 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:04.010062 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:04.010063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:04.010064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:04.010131 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:04.010531 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:04.010532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:04.010532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:04.010646 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:04.010699 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.010964 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.011035 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:20:04.011268 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.001240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41247 (local address 127.15.216.67:42827)
0810 04:20:04.001431 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.001477 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.002140 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.002480 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.002484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.003207 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:04.003431 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.003436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.004111 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.004111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.004986 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.004988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.005345 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.005345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.005455 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.006079 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:04.006087 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:04.007614 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:04.009597 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:04.009605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:04.009608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:04.009972 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:04.010227 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:04.010228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:04.010229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:04.010434 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:04.010705 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:04.010711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:04.010834 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.010919 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.011052 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260810 04:20:04.011708 2475 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41247
I20260810 04:20:04.011794 2475 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:04.011955 2475 heartbeater.cc:511] Master 127.15.216.126:41247 requested a full tablet report, sending...
I20260810 04:20:04.012260 1973 ts_manager.cc:194] Registered new tserver with Master: 7212eda7ff3a4f278b7afb67fb9a30ec (127.15.216.67:40079)
I20260810 04:20:04.012660 1973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:42827
I20260810 04:20:04.021507 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:20:04.029626 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.023011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.15.216.126:41247)
0810 04:20:04.023106 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.023109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.023236 (+ 127us) server_negotiation.cc:407] Connection header received
0810 04:20:04.023266 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.023267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.023282 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.023349 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:04.024071 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.024560 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.025326 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.025522 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.025625 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.026957 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:04.026959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:04.026959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:04.026967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:04.028385 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:04.028752 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:04.028753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:04.028754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:04.028812 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:04.029021 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:04.029022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:04.029022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:04.029177 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:04.029291 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.029399 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.029481 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 04:20:04.031349 1973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:20:04.032848 1973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:20:04.039898 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.035209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:42739 (local address 127.0.0.1:48874)
0810 04:20:04.035352 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.035394 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.035465 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.035972 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.035976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.035979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.036061 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.036063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.037230 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.037231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.038491 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.038494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.039573 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.039574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.039670 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.039686 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.039755 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.039813 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":53,"threads_started":1}
I20260810 04:20:04.040102 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.035237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38179 (local address 127.0.0.1:44778)
0810 04:20:04.035718 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.035730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.035786 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.036464 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.036465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.036467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.036512 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.036513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.038127 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.038128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.039029 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.039031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.039791 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.039792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.039864 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.039868 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.039942 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.039988 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":48,"thread_start_us":98,"threads_started":1}
I20260810 04:20:04.040112 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.035706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.15.216.66:38179)
0810 04:20:04.035903 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.035930 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.035940 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:20:04.036123 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.036125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.036142 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.036225 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.036629 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.038019 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.039121 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.039687 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.039771 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.039825 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.039961 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.040014 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260810 04:20:04.040294 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.035494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.15.216.65:42739)
0810 04:20:04.035725 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.035730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.035744 (+ 14us) server_negotiation.cc:407] Connection header received
0810 04:20:04.035814 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.035816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.035838 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.035961 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.036150 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.037111 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.038602 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.039459 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.039561 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.039635 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.040121 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.040172 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260810 04:20:04.040902 2110 tablet_service.cc:1467] Processing CreateTablet for tablet 71c49dda19fa4355b2a432f970538cfa (DEFAULT_TABLE table=test-table [id=fca1c7ae6f8d4837a7dadbab6a7d760b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:04.041007 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 71c49dda19fa4355b2a432f970538cfa (DEFAULT_TABLE table=test-table [id=fca1c7ae6f8d4837a7dadbab6a7d760b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:04.041325 2110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c49dda19fa4355b2a432f970538cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:04.041440 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c49dda19fa4355b2a432f970538cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:04.041859 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.035526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.15.216.67:40079)
0810 04:20:04.035752 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.035756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.035981 (+ 225us) server_negotiation.cc:407] Connection header received
0810 04:20:04.036102 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.036105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.036121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.036178 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.036382 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.037849 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.039670 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.041245 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.041309 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.041368 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.041697 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.041752 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260810 04:20:04.042106 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.035431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:40079 (local address 127.0.0.1:39512)
0810 04:20:04.035896 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.035906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.036009 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.036279 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.036279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.036281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.036317 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.036318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.038267 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.038268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.039536 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.039539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.041453 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.041454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.041529 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.041533 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.041961 (+ 428us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.042024 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":89,"threads_started":1}
I20260810 04:20:04.044381 2495 tablet_bootstrap.cc:492] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: Bootstrap starting.
I20260810 04:20:04.044747 2496 tablet_bootstrap.cc:492] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Bootstrap starting.
I20260810 04:20:04.045205 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 71c49dda19fa4355b2a432f970538cfa (DEFAULT_TABLE table=test-table [id=fca1c7ae6f8d4837a7dadbab6a7d760b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:04.045485 2496 tablet_bootstrap.cc:654] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:04.045486 2495 tablet_bootstrap.cc:654] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:04.045607 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c49dda19fa4355b2a432f970538cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:04.046185 2495 log.cc:824] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:04.046185 2496 log.cc:824] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:04.047076 2495 tablet_bootstrap.cc:492] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: No bootstrap required, opened a new log
I20260810 04:20:04.047158 2495 ts_tablet_manager.cc:1397] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:20:04.047528 2496 tablet_bootstrap.cc:492] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: No bootstrap required, opened a new log
I20260810 04:20:04.047605 2496 ts_tablet_manager.cc:1397] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:20:04.049127 2499 tablet_bootstrap.cc:492] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Bootstrap starting.
I20260810 04:20:04.049293 2495 raft_consensus.cc:348] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.049525 2495 raft_consensus.cc:374] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:04.049621 2495 raft_consensus.cc:729] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf09d3202ce8484d9b8a8b4eeeb5ef9a, State: Initialized, Role: FOLLOWER
I20260810 04:20:04.049809 2495 consensus_queue.cc:260] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.049896 2496 raft_consensus.cc:348] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.050072 2496 raft_consensus.cc:374] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:04.050122 2495 ts_tablet_manager.cc:1428] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 04:20:04.050119 2496 raft_consensus.cc:729] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8e951d2489435fb7422c20f4b41d9c, State: Initialized, Role: FOLLOWER
I20260810 04:20:04.050166 2499 tablet_bootstrap.cc:654] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:04.050243 2179 heartbeater.cc:503] Master 127.15.216.126:41247 was elected leader, sending a full tablet report...
I20260810 04:20:04.050271 2496 consensus_queue.cc:260] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.050575 2326 heartbeater.cc:503] Master 127.15.216.126:41247 was elected leader, sending a full tablet report...
I20260810 04:20:04.050788 2496 ts_tablet_manager.cc:1428] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:20:04.050880 2499 log.cc:824] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:04.051756 2499 tablet_bootstrap.cc:492] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: No bootstrap required, opened a new log
I20260810 04:20:04.051815 2499 ts_tablet_manager.cc:1397] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:20:04.053272 2499 raft_consensus.cc:348] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.053402 2499 raft_consensus.cc:374] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:04.053422 2499 raft_consensus.cc:729] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7212eda7ff3a4f278b7afb67fb9a30ec, State: Initialized, Role: FOLLOWER
I20260810 04:20:04.053536 2499 consensus_queue.cc:260] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.053820 2499 ts_tablet_manager.cc:1428] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:20:04.053905 2475 heartbeater.cc:503] Master 127.15.216.126:41247 was elected leader, sending a full tablet report...
W20260810 04:20:04.071475 2327 tablet.cc:2369] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:04.154495 2180 tablet.cc:2369] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:04.251204 2476 tablet.cc:2369] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:20:04.261771 2500 raft_consensus.cc:482] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:04.261917 2500 raft_consensus.cc:504] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.262239 2500 leader_election.cc:288] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7212eda7ff3a4f278b7afb67fb9a30ec (127.15.216.67:40079), 3e8e951d2489435fb7422c20f4b41d9c (127.15.216.66:38179)
I20260810 04:20:04.265993 2503 raft_consensus.cc:482] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:04.266141 2503 raft_consensus.cc:504] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.266577 2503 leader_election.cc:288] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf09d3202ce8484d9b8a8b4eeeb5ef9a (127.15.216.65:42739), 3e8e951d2489435fb7422c20f4b41d9c (127.15.216.66:38179)
I20260810 04:20:04.267350 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.262591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:58607 (local address 127.15.216.67:40079)
0810 04:20:04.262642 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.262644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.262675 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:20:04.262793 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.262794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.262805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.262887 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.263263 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.264061 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.265997 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.266633 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.266712 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.266750 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.267222 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.267270 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 04:20:04.267352 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.262592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:57271 (local address 127.15.216.66:38179)
0810 04:20:04.262642 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.262644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.262678 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:20:04.262793 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.262794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.262808 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.262887 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.263210 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.264082 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.266057 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.266827 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.266912 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.266925 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.267182 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.267236 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:20:04.267632 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.262472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38179 (local address 127.15.216.65:57271)
0810 04:20:04.262610 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.262623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.262695 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.262943 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.262944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.262948 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.263052 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.263054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.264211 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.264213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.265898 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.265902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.266967 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.266969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.267073 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.267079 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.267486 (+ 407us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.267535 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":164,"thread_start_us":65,"threads_started":1}
I20260810 04:20:04.267851 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.262473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:40079 (local address 127.15.216.65:58607)
0810 04:20:04.262610 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.262623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.262713 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.262943 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.262944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.262947 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.263058 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.263059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.264213 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.264214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.265898 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.265902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.267042 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.267043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.267131 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.267137 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.267688 (+ 551us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.267745 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":66,"thread_start_us":57,"threads_started":1}
I20260810 04:20:04.268330 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c49dda19fa4355b2a432f970538cfa" candidate_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" is_pre_election: true
I20260810 04:20:04.268407 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c49dda19fa4355b2a432f970538cfa" candidate_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8e951d2489435fb7422c20f4b41d9c" is_pre_election: true
I20260810 04:20:04.268457 2424 raft_consensus.cc:2454] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf09d3202ce8484d9b8a8b4eeeb5ef9a in term 0.
I20260810 04:20:04.268556 2275 raft_consensus.cc:2454] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf09d3202ce8484d9b8a8b4eeeb5ef9a in term 0.
I20260810 04:20:04.268724 2057 leader_election.cc:302] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7212eda7ff3a4f278b7afb67fb9a30ec, cf09d3202ce8484d9b8a8b4eeeb5ef9a; no voters:
I20260810 04:20:04.268973 2500 raft_consensus.cc:2781] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:20:04.269114 2500 raft_consensus.cc:482] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:20:04.269192 2500 raft_consensus.cc:3037] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:04.270170 2500 raft_consensus.cc:504] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.270402 2500 leader_election.cc:288] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [CANDIDATE]: Term 1 election: Requested vote from peers 7212eda7ff3a4f278b7afb67fb9a30ec (127.15.216.67:40079), 3e8e951d2489435fb7422c20f4b41d9c (127.15.216.66:38179)
I20260810 04:20:04.270493 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c49dda19fa4355b2a432f970538cfa" candidate_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8e951d2489435fb7422c20f4b41d9c"
I20260810 04:20:04.270524 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c49dda19fa4355b2a432f970538cfa" candidate_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec"
I20260810 04:20:04.270572 2275 raft_consensus.cc:3037] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:04.270612 2424 raft_consensus.cc:3037] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:04.270845 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.266755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:42739 (local address 127.15.216.67:56401)
0810 04:20:04.266900 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.266912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.266955 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.267222 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.267223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.267225 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.267283 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.267284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.268283 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.268284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.269480 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.269483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.270529 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.270530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.270615 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.270619 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.270676 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.270726 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":2}
I20260810 04:20:04.271013 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.266947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:60691 (local address 127.15.216.66:38179)
0810 04:20:04.267462 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.267464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.267472 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:04.267497 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.267498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.267509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.267575 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.267752 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.268465 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.269788 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.270472 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.270547 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.270556 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.270887 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.270933 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":84,"threads_started":1}
I20260810 04:20:04.271113 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.267035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:56401 (local address 127.15.216.65:42739)
0810 04:20:04.267080 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.267082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.267089 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:20:04.267107 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.267108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.267120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.267177 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.267374 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.268167 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.269613 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.270262 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.270341 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.270350 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.271015 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.271054 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:20:04.271230 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.266864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38179 (local address 127.15.216.67:60691)
0810 04:20:04.267110 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.267120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.267166 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.267612 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.267613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.267615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.267679 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.267681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.268555 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.268556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.269682 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.269685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.270592 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.270593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.270677 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.270681 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.270742 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.270794 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":119,"threads_started":1}
I20260810 04:20:04.271397 2132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c49dda19fa4355b2a432f970538cfa" candidate_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" is_pre_election: true
I20260810 04:20:04.271534 2132 raft_consensus.cc:2379] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7212eda7ff3a4f278b7afb67fb9a30ec in current term 1: Already voted for candidate cf09d3202ce8484d9b8a8b4eeeb5ef9a in this term.
I20260810 04:20:04.271710 2275 raft_consensus.cc:2454] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf09d3202ce8484d9b8a8b4eeeb5ef9a in term 1.
I20260810 04:20:04.271726 2424 raft_consensus.cc:2454] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf09d3202ce8484d9b8a8b4eeeb5ef9a in term 1.
I20260810 04:20:04.271899 2058 leader_election.cc:302] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8e951d2489435fb7422c20f4b41d9c, cf09d3202ce8484d9b8a8b4eeeb5ef9a; no voters:
I20260810 04:20:04.271867 2274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c49dda19fa4355b2a432f970538cfa" candidate_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8e951d2489435fb7422c20f4b41d9c" is_pre_election: true
I20260810 04:20:04.271971 2274 raft_consensus.cc:2379] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7212eda7ff3a4f278b7afb67fb9a30ec in current term 1: Already voted for candidate cf09d3202ce8484d9b8a8b4eeeb5ef9a in this term.
I20260810 04:20:04.272024 2500 raft_consensus.cc:2781] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:04.272204 2500 raft_consensus.cc:686] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 1 LEADER]: Becoming Leader. State: Replica: cf09d3202ce8484d9b8a8b4eeeb5ef9a, State: Running, Role: LEADER
I20260810 04:20:04.272239 2352 leader_election.cc:302] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7212eda7ff3a4f278b7afb67fb9a30ec; no voters: 3e8e951d2489435fb7422c20f4b41d9c, cf09d3202ce8484d9b8a8b4eeeb5ef9a
I20260810 04:20:04.272348 2500 consensus_queue.cc:237] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } }
I20260810 04:20:04.272398 2503 raft_consensus.cc:2726] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 04:20:04.273371 1973 catalog_manager.cc:5495] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a reported cstate change: term changed from 0 to 1, leader changed from <none> to cf09d3202ce8484d9b8a8b4eeeb5ef9a (127.15.216.65). New cstate: current_term: 1 leader_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf09d3202ce8484d9b8a8b4eeeb5ef9a" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 42739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 } health_report { overall_health: UNKNOWN } } }
I20260810 04:20:04.330483 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.327768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.15.216.65:42739)
0810 04:20:04.327856 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.327858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.327869 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:20:04.327931 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.327933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.327960 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.328046 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:04.328220 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.328716 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.329530 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.329762 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.329824 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.330037 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:04.330156 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:04.330294 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.330332 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.330353 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 04:20:04.332513 2274 raft_consensus.cc:1261] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 1 FOLLOWER]: Refusing update from remote peer cf09d3202ce8484d9b8a8b4eeeb5ef9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:04.332593 2424 raft_consensus.cc:1261] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 1 FOLLOWER]: Refusing update from remote peer cf09d3202ce8484d9b8a8b4eeeb5ef9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:04.332881 2500 consensus_queue.cc:1035] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8e951d2489435fb7422c20f4b41d9c" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:04.333053 2509 consensus_queue.cc:1035] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7212eda7ff3a4f278b7afb67fb9a30ec" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:04.335492 2512 mvcc.cc:205] Tried to move back new op lower bound from 7316830635342786560 to 7316830635100704768. Current Snapshot: MvccSnapshot[applied={T|T < 7316830635342786560}]
I20260810 04:20:04.337996 1972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:04.338117 1972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:04.338876 1972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6b4756c9acd415491ce78f178661fcd: Sending DeleteTablet for 3 replicas of tablet 71c49dda19fa4355b2a432f970538cfa
I20260810 04:20:04.339227 2110 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c49dda19fa4355b2a432f970538cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:04 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:48874
I20260810 04:20:04.339251 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c49dda19fa4355b2a432f970538cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:04 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:39512
I20260810 04:20:04.339479 2523 tablet_replica.cc:330] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: stopping tablet replica
I20260810 04:20:04.339479 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 71c49dda19fa4355b2a432f970538cfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:04 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:44778
I20260810 04:20:04.339605 2523 raft_consensus.cc:2229] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:20:04.339711 2523 raft_consensus.cc:2258] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:04.339742 2525 tablet_replica.cc:330] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: stopping tablet replica
I20260810 04:20:04.339838 2525 raft_consensus.cc:2229] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:04.339933 2525 raft_consensus.cc:2258] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:04.340056 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 2040
I20260810 04:20:04.340188 2523 ts_tablet_manager.cc:1905] T 71c49dda19fa4355b2a432f970538cfa P cf09d3202ce8484d9b8a8b4eeeb5ef9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:04.340761 2525 ts_tablet_manager.cc:1905] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:04.342597 2525 ts_tablet_manager.cc:1918] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:04.342669 2525 log.cc:1196] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/71c49dda19fa4355b2a432f970538cfa
I20260810 04:20:04.342936 2525 ts_tablet_manager.cc:1939] T 71c49dda19fa4355b2a432f970538cfa P 3e8e951d2489435fb7422c20f4b41d9c: Deleting consensus metadata
I20260810 04:20:04.343359 1955 catalog_manager.cc:4841] TS 3e8e951d2489435fb7422c20f4b41d9c (127.15.216.66:38179): tablet 71c49dda19fa4355b2a432f970538cfa (table test-table [id=fca1c7ae6f8d4837a7dadbab6a7d760b]) successfully deleted
I20260810 04:20:04.345744 2524 tablet_replica.cc:330] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: stopping tablet replica
I20260810 04:20:04.345908 2524 raft_consensus.cc:2229] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:04.346050 2524 raft_consensus.cc:2258] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:04.346578 2524 ts_tablet_manager.cc:1905] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:04.348043 2524 ts_tablet_manager.cc:1918] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260810 04:20:04.348039 1954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:42739 (error 108)
I20260810 04:20:04.348129 2524 log.cc:1196] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/71c49dda19fa4355b2a432f970538cfa
I20260810 04:20:04.348453 2524 ts_tablet_manager.cc:1939] T 71c49dda19fa4355b2a432f970538cfa P 7212eda7ff3a4f278b7afb67fb9a30ec: Deleting consensus metadata
I20260810 04:20:04.348644 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.348477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:42739 (local address 127.0.0.1:48898)
0810 04:20:04.348538 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.348578 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:42739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:20:04.348716 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 2184
W20260810 04:20:04.351936 1954 catalog_manager.cc:4584] TS cf09d3202ce8484d9b8a8b4eeeb5ef9a (127.15.216.65:42739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71c49dda19fa4355b2a432f970538cfa: Network error: Client connection negotiation failed: client connection to 127.15.216.65:42739: connect: Connection refused (error 111)
I20260810 04:20:04.352079 1954 catalog_manager.cc:4841] TS 7212eda7ff3a4f278b7afb67fb9a30ec (127.15.216.67:40079): tablet 71c49dda19fa4355b2a432f970538cfa (table test-table [id=fca1c7ae6f8d4837a7dadbab6a7d760b]) successfully deleted
I20260810 04:20:04.356701 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 2333
I20260810 04:20:04.363524 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 1938
2026-08-10T04:20:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2269 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T04:20:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:04Z Disabled control of system clock
I20260810 04:20:04.395061 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36841
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:36841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:20:04.478332 2530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:04.478577 2530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:04.478600 2530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:04.479794 2530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:04.479840 2530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:04.479853 2530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:04.479866 2530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:04.479878 2530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:04.481195 2530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:36841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:04.481402 2530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:04.481721 2530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:04.486892 2535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:04.486922 2539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:04.486909 2537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:04.486997 2530 server_base.cc:1034] running on GCE node
I20260810 04:20:04.487561 2530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:04.487867 2530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:04.489051 2530 hybrid_clock.cc:648] HybridClock initialized: now 1786335604489026 us; error 60 us; skew 500 ppm
I20260810 04:20:04.490419 2530 webserver.cc:460] Webserver started at http://127.15.216.126:41415/ using document root <none> and password file <none>
I20260810 04:20:04.490675 2530 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:04.490768 2530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:04.490924 2530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:04.491994 2530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "392994a62e4bd75892251a90b78a9be5"
server_key_iv: "fe3a73ee230b8d595bdbba3e53218b18"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.492372 2530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "392994a62e4bd75892251a90b78a9be5"
server_key_iv: "fe3a73ee230b8d595bdbba3e53218b18"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.493827 2530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:04.494679 2545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.494874 2530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:04.494993 2530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "392994a62e4bd75892251a90b78a9be5"
server_key_iv: "fe3a73ee230b8d595bdbba3e53218b18"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.495074 2530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:04.504302 2530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:04.504626 2530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:04.504787 2530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:04.509114 2530 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:36841
I20260810 04:20:04.509150 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:36841 every 8 connection(s)
I20260810 04:20:04.509503 2530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:04.509864 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 2530
I20260810 04:20:04.509987 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:04.510216 16225 external_mini_cluster.cc:1428] Setting key 1303be8c0461fd72b80f30ba9da0b1cf
I20260810 04:20:04.510773 2601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:04.514221 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c: Bootstrap starting.
I20260810 04:20:04.514839 2601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:04.515141 2601 log.cc:824] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:04.515974 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c: No bootstrap required, opened a new log
I20260810 04:20:04.516577 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.510879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.15.216.126:36841)
0810 04:20:04.511120 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.511128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.511163 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:20:04.511686 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.511694 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.511776 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.511927 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:04.512132 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.513461 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.514412 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.514848 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.515012 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.515848 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:04.515852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:04.515853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:04.516052 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:04.516119 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:04.516135 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:04.516211 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.516304 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.516387 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260810 04:20:04.518254 2601 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36841 } }
I20260810 04:20:04.518409 2601 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:04.518456 2601 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6b4c14d3e94aa2bef91bb7fd1f287c, State: Initialized, Role: FOLLOWER
I20260810 04:20:04.518595 2601 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36841 } }
I20260810 04:20:04.518678 2601 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:04.518723 2601 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:04.518779 2601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:04.519306 2601 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36841 } }
I20260810 04:20:04.519428 2601 leader_election.cc:302] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa6b4c14d3e94aa2bef91bb7fd1f287c; no voters:
I20260810 04:20:04.519610 2601 leader_election.cc:288] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:04.519724 2606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:04.519893 2601 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:04.519901 2606 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [term 1 LEADER]: Becoming Leader. State: Replica: fa6b4c14d3e94aa2bef91bb7fd1f287c, State: Running, Role: LEADER
I20260810 04:20:04.520092 2606 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36841 } }
I20260810 04:20:04.520478 2608 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa6b4c14d3e94aa2bef91bb7fd1f287c. Latest consensus state: current_term: 1 leader_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36841 } } }
I20260810 04:20:04.520577 2608 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:04.520448 2607 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa6b4c14d3e94aa2bef91bb7fd1f287c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36841 } } }
I20260810 04:20:04.520711 2607 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:04.521418 2617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:04.522063 2617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:04.523535 2617 catalog_manager.cc:1347] Generated new cluster ID: 28836f2b4f554c8bbc7e756c123ad150
I20260810 04:20:04.523587 2617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:04.532101 2617 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:04.532629 2617 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:04.536670 2617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c: Generated new TSK 0
I20260810 04:20:04.536868 2617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:04.538578 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36841
--builtin_ntp_servers=127.15.216.84:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:20:04.624333 2626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:04.624589 2626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:04.624632 2626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:04.626083 2626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:04.626153 2626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:04.626202 2626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:20:04.627724 2626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:04.628015 2626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:04.628355 2626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:04.634660 2631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:04.634692 2632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:04.634804 2634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:04.634990 2626 server_base.cc:1034] running on GCE node
I20260810 04:20:04.635236 2626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:04.635485 2626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:04.636652 2626 hybrid_clock.cc:648] HybridClock initialized: now 1786335604636639 us; error 26 us; skew 500 ppm
I20260810 04:20:04.638216 2626 webserver.cc:460] Webserver started at http://127.15.216.65:33933/ using document root <none> and password file <none>
I20260810 04:20:04.638479 2626 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:04.638538 2626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:04.638670 2626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:04.640084 2626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "c7a2e6487d474f61806cf3fe82dc76fa"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "5fd759049ed829e8885ef4b97f0ad934"
server_key_iv: "b3a4c71d46fcbc3221db448f97723a95"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.640479 2626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "c7a2e6487d474f61806cf3fe82dc76fa"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "5fd759049ed829e8885ef4b97f0ad934"
server_key_iv: "b3a4c71d46fcbc3221db448f97723a95"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.642165 2626 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 04:20:04.643081 2640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.643288 2626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:04.643388 2626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "c7a2e6487d474f61806cf3fe82dc76fa"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "5fd759049ed829e8885ef4b97f0ad934"
server_key_iv: "b3a4c71d46fcbc3221db448f97723a95"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.643460 2626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:04.669726 2626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:04.670032 2626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:04.670140 2626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:04.670337 2626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:04.670724 2626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:04.670763 2626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.670818 2626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:04.670854 2626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.677139 2626 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:36737
I20260810 04:20:04.677196 2766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:36737 every 8 connection(s)
I20260810 04:20:04.677541 2626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:20:04.683234 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.678320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:43947 (local address 127.15.216.126:36841)
0810 04:20:04.678473 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.678475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.678597 (+ 122us) server_negotiation.cc:407] Connection header received
0810 04:20:04.679071 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.679072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.679087 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.679146 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:04.679863 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.680497 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.681471 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.681788 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.681857 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.682937 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:04.682938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:04.682939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:04.682943 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:04.682957 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:04.682962 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:04.682988 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.683108 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.683151 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260810 04:20:04.683678 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.678271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36841 (local address 127.15.216.65:43947)
0810 04:20:04.678467 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.678503 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.678984 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.679224 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.679228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.679236 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:04.679746 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.679750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.680594 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.680595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.681372 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.681374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.681880 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.681881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.682004 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.682745 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:04.682746 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:04.682790 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:04.682818 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:04.683027 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:04.683033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:04.683036 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.683378 (+ 342us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.683501 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260810 04:20:04.684346 2767 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36841
I20260810 04:20:04.684484 2767 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:04.684739 2767 heartbeater.cc:511] Master 127.15.216.126:36841 requested a full tablet report, sending...
I20260810 04:20:04.684798 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 2626
I20260810 04:20:04.684888 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:20:04.685125 16225 external_mini_cluster.cc:1428] Setting key 75fd732eb4f203c2a274de935520f31e
I20260810 04:20:04.685402 2562 ts_manager.cc:194] Registered new tserver with Master: c7a2e6487d474f61806cf3fe82dc76fa (127.15.216.65:36737)
I20260810 04:20:04.686237 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36841
--builtin_ntp_servers=127.15.216.84:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:20:04.686259 2562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.65:43947
W20260810 04:20:04.774647 2770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:04.774909 2770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:04.774950 2770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:04.776144 2770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:04.776227 2770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:04.776273 2770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:20:04.777755 2770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:04.778101 2770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:04.778445 2770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:04.783946 2777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:04.784075 2779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:04.783970 2776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:04.784219 2770 server_base.cc:1034] running on GCE node
I20260810 04:20:04.784505 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:04.784766 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:04.785934 2770 hybrid_clock.cc:648] HybridClock initialized: now 1786335604785933 us; error 42 us; skew 500 ppm
I20260810 04:20:04.787343 2770 webserver.cc:460] Webserver started at http://127.15.216.66:41321/ using document root <none> and password file <none>
I20260810 04:20:04.787561 2770 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:04.787603 2770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:04.787693 2770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:04.788761 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "c294c5ac55ebe4042156b10f069959ac"
server_key_iv: "1909f7629fe2bcce7013c056e2d0eea0"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.789065 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "c294c5ac55ebe4042156b10f069959ac"
server_key_iv: "1909f7629fe2bcce7013c056e2d0eea0"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.790572 2770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:04.791428 2785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.791770 2770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:04.791901 2770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "c294c5ac55ebe4042156b10f069959ac"
server_key_iv: "1909f7629fe2bcce7013c056e2d0eea0"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.791989 2770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:04.808650 2770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:04.809016 2770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:04.809190 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:04.809509 2770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:04.809988 2770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:04.810055 2770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.810103 2770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:04.810132 2770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.816438 2770 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:37567
I20260810 04:20:04.816577 2906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:37567 every 8 connection(s)
I20260810 04:20:04.816877 2770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:20:04.822389 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 2770
I20260810 04:20:04.822482 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:20:04.822695 16225 external_mini_cluster.cc:1428] Setting key e8beef867fc1ce2e0b7c9b252cb37386
I20260810 04:20:04.822918 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.817856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:35233 (local address 127.15.216.126:36841)
0810 04:20:04.817994 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.817996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.818176 (+ 180us) server_negotiation.cc:407] Connection header received
0810 04:20:04.818660 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.818661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.818672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.818739 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:04.819411 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.819943 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.821225 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.821431 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.821496 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.822579 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:04.822580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:04.822581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:04.822585 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:04.822602 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:04.822606 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:04.822662 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.822809 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.822837 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 04:20:04.823348 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.817839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36841 (local address 127.15.216.66:35233)
0810 04:20:04.818031 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.818074 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.818569 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.818791 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.818794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.818802 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:04.819319 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.819328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.820099 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.820100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.821151 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.821154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.821539 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.821540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.821687 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.822382 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:04.822382 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:04.822413 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:04.822460 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:04.822673 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:04.822679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:04.822682 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.822776 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.822887 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260810 04:20:04.823737 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36841
--builtin_ntp_servers=127.15.216.84:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:20:04.824091 2907 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36841
I20260810 04:20:04.824201 2907 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:04.824393 2907 heartbeater.cc:511] Master 127.15.216.126:36841 requested a full tablet report, sending...
I20260810 04:20:04.824824 2562 ts_manager.cc:194] Registered new tserver with Master: 8fde6a5c3f1547d6a7dc325b4a03c4a7 (127.15.216.66:37567)
I20260810 04:20:04.825217 2562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.66:35233
W20260810 04:20:04.910176 2911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:04.910380 2911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:04.910403 2911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:04.911550 2911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:04.911597 2911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:04.911623 2911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:20:04.913015 2911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:36841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:04.913300 2911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:04.913646 2911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:04.919139 2917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:04.919236 2916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:04.919157 2911 server_base.cc:1034] running on GCE node
W20260810 04:20:04.919135 2919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:04.919618 2911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:04.919910 2911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:04.921176 2911 hybrid_clock.cc:648] HybridClock initialized: now 1786335604921145 us; error 43 us; skew 500 ppm
I20260810 04:20:04.922652 2911 webserver.cc:460] Webserver started at http://127.15.216.67:39419/ using document root <none> and password file <none>
I20260810 04:20:04.922957 2911 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:04.923038 2911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:04.923209 2911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:04.924346 2911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "e415643d2e3a4216949a53e473893b75"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "65744ac774edbaf5f09d41bb053f1e2b"
server_key_iv: "d54e4a9e651e94b998d8b061d3700984"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.924736 2911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "e415643d2e3a4216949a53e473893b75"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "65744ac774edbaf5f09d41bb053f1e2b"
server_key_iv: "d54e4a9e651e94b998d8b061d3700984"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.926335 2911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260810 04:20:04.927193 2926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.927500 2911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:04.927625 2911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "e415643d2e3a4216949a53e473893b75"
format_stamp: "Formatted at 2026-08-10 04:20:04 on dist-test-slave-g350"
server_key: "65744ac774edbaf5f09d41bb053f1e2b"
server_key_iv: "d54e4a9e651e94b998d8b061d3700984"
server_key_version: "encryptionkey@0"
I20260810 04:20:04.927713 2911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:04.939438 2911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:04.939769 2911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:04.939936 2911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:04.940181 2911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:04.940541 2911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:04.940608 2911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.940650 2911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:04.940678 2911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:04.947063 2911 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:34069
I20260810 04:20:04.947115 3055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:34069 every 8 connection(s)
I20260810 04:20:04.947522 2911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:20:04.950012 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 2911
I20260810 04:20:04.950129 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:20:04.950348 16225 external_mini_cluster.cc:1428] Setting key 4f5e60ed5ec790dfdab76b912f153401
I20260810 04:20:04.953567 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.948274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:46395 (local address 127.15.216.126:36841)
0810 04:20:04.948370 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.948373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.948671 (+ 298us) server_negotiation.cc:407] Connection header received
0810 04:20:04.949173 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.949174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.949185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.949242 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:04.950015 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.950746 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.952065 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.952263 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.952361 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.953253 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:04.953254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:04.953254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:04.953258 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:04.953309 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:04.953314 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:04.953361 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.953478 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.953502 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 04:20:04.953778 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.948247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:36841 (local address 127.15.216.67:46395)
0810 04:20:04.948527 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.948572 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.949071 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.949292 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.949295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.949303 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:04.949916 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.949923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.950883 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.950884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.951978 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.951980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.952396 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.952397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.952506 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.953113 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:04.953114 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:04.953143 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:04.953165 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:04.953379 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:04.953382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:04.953384 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.953433 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.953509 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260810 04:20:04.954367 3056 heartbeater.cc:347] Connected to a master server at 127.15.216.126:36841
I20260810 04:20:04.954488 3056 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:04.954680 3056 heartbeater.cc:511] Master 127.15.216.126:36841 requested a full tablet report, sending...
I20260810 04:20:04.955060 2562 ts_manager.cc:194] Registered new tserver with Master: e415643d2e3a4216949a53e473893b75 (127.15.216.67:34069)
I20260810 04:20:04.955523 2562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.67:46395
I20260810 04:20:04.962556 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:20:04.967370 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.964069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.15.216.126:36841)
0810 04:20:04.964176 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.964178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.964344 (+ 166us) server_negotiation.cc:407] Connection header received
0810 04:20:04.964453 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.964454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.964466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.964519 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:04.964694 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.965183 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.966045 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.966239 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.966318 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.967073 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:04.967075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:04.967076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:04.967081 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:04.967098 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:04.967102 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:04.967184 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.967256 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.967279 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 04:20:04.969141 2562 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:20:04.970901 2562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:20:04.977295 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.973250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.15.216.66:37567)
0810 04:20:04.973680 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.973684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.973696 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:20:04.973738 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.973741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.973753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.973849 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.974014 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.974840 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.976263 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.976902 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.976994 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.977086 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.977173 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.977211 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":112,"threads_started":1}
I20260810 04:20:04.977285 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.973137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:37567 (local address 127.0.0.1:44828)
0810 04:20:04.973466 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.973501 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.973562 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.973855 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.973858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.973860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.973929 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.973930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.974933 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.974934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.976164 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.976167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.976992 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.976993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.977067 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.977080 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.977157 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.977209 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":152,"threads_started":1}
I20260810 04:20:04.977730 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.973278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34069 (local address 127.0.0.1:57926)
0810 04:20:04.973484 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.973494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.973561 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.973975 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.973976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.973978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.974024 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.974026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.975254 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.975255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.976315 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.976317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.977311 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.977312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.977512 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.977516 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.977577 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.977652 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":8,"thread_start_us":121,"threads_started":1}
I20260810 04:20:04.977762 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.973542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.15.216.67:34069)
0810 04:20:04.973762 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.973766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.973777 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:20:04.973817 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.973819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.973834 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.973908 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.974101 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.975157 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.976400 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.977101 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.977174 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.977268 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.977633 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.977681 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":115,"threads_started":1}
I20260810 04:20:04.977847 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.973197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.15.216.65:36737)
0810 04:20:04.973430 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:04.973436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:04.973562 (+ 126us) server_negotiation.cc:407] Connection header received
0810 04:20:04.974043 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:04.974047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:04.974065 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:04.974116 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:04.974345 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.975391 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.976575 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.977297 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.977363 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.977461 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:04.977698 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 04:20:04.977749 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20260810 04:20:04.978019 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:04.973069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:36737 (local address 127.0.0.1:39676)
0810 04:20:04.973437 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:04.973492 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:04.973919 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:04.974206 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:04.974207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:04.974209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:04.974255 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.974256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.975456 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.975457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.976475 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:04.976478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:04.977492 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:04.977492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:04.977546 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:04.977548 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:20:04.977620 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 04:20:04.977656 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260810 04:20:04.978288 2839 tablet_service.cc:1467] Processing CreateTablet for tablet e64e5d53a7384045b32e1f17493ee49f (DEFAULT_TABLE table=test-table [id=768b7677f5734514a497edb896d62d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:04.978615 2980 tablet_service.cc:1467] Processing CreateTablet for tablet e64e5d53a7384045b32e1f17493ee49f (DEFAULT_TABLE table=test-table [id=768b7677f5734514a497edb896d62d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:04.978693 2839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64e5d53a7384045b32e1f17493ee49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:04.978632 2698 tablet_service.cc:1467] Processing CreateTablet for tablet e64e5d53a7384045b32e1f17493ee49f (DEFAULT_TABLE table=test-table [id=768b7677f5734514a497edb896d62d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:04.978927 2980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64e5d53a7384045b32e1f17493ee49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:04.978953 2698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64e5d53a7384045b32e1f17493ee49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:04.981541 3071 tablet_bootstrap.cc:492] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Bootstrap starting.
I20260810 04:20:04.982213 3072 tablet_bootstrap.cc:492] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Bootstrap starting.
I20260810 04:20:04.982333 3071 tablet_bootstrap.cc:654] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:04.982703 3071 log.cc:824] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:04.982897 3072 tablet_bootstrap.cc:654] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:04.983230 3072 log.cc:824] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:04.983531 3071 tablet_bootstrap.cc:492] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: No bootstrap required, opened a new log
I20260810 04:20:04.983582 3073 tablet_bootstrap.cc:492] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: Bootstrap starting.
I20260810 04:20:04.983606 3071 ts_tablet_manager.cc:1397] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:20:04.983999 3072 tablet_bootstrap.cc:492] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: No bootstrap required, opened a new log
I20260810 04:20:04.984071 3072 ts_tablet_manager.cc:1397] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:20:04.986112 3073 tablet_bootstrap.cc:654] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:04.986430 3073 log.cc:824] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:04.986428 3071 raft_consensus.cc:348] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:04.986614 3071 raft_consensus.cc:374] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:04.986665 3071 raft_consensus.cc:729] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e415643d2e3a4216949a53e473893b75, State: Initialized, Role: FOLLOWER
I20260810 04:20:04.986641 3072 raft_consensus.cc:348] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:04.986799 3072 raft_consensus.cc:374] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:04.986843 3072 raft_consensus.cc:729] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fde6a5c3f1547d6a7dc325b4a03c4a7, State: Initialized, Role: FOLLOWER
I20260810 04:20:04.986804 3071 consensus_queue.cc:260] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:04.986955 3072 consensus_queue.cc:260] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:04.987087 3071 ts_tablet_manager.cc:1428] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 04:20:04.987207 3072 ts_tablet_manager.cc:1428] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:20:04.987210 3056 heartbeater.cc:503] Master 127.15.216.126:36841 was elected leader, sending a full tablet report...
I20260810 04:20:04.987406 3073 tablet_bootstrap.cc:492] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: No bootstrap required, opened a new log
I20260810 04:20:04.987491 3073 ts_tablet_manager.cc:1397] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:20:04.987485 2907 heartbeater.cc:503] Master 127.15.216.126:36841 was elected leader, sending a full tablet report...
I20260810 04:20:04.990098 3073 raft_consensus.cc:348] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:04.990245 3073 raft_consensus.cc:374] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:04.990293 3073 raft_consensus.cc:729] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a2e6487d474f61806cf3fe82dc76fa, State: Initialized, Role: FOLLOWER
I20260810 04:20:04.990406 3073 consensus_queue.cc:260] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:04.990684 3073 ts_tablet_manager.cc:1428] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:20:04.990806 2767 heartbeater.cc:503] Master 127.15.216.126:36841 was elected leader, sending a full tablet report...
W20260810 04:20:05.067546 2908 tablet.cc:2369] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:05.178277 2768 tablet.cc:2369] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:05.198107 3058 tablet.cc:2369] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:20:05.205276 3082 raft_consensus.cc:482] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:05.205413 3082 raft_consensus.cc:504] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:05.205840 3082 leader_election.cc:288] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fde6a5c3f1547d6a7dc325b4a03c4a7 (127.15.216.66:37567), e415643d2e3a4216949a53e473893b75 (127.15.216.67:34069)
I20260810 04:20:05.209955 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:05.206124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:60203 (local address 127.15.216.67:34069)
0810 04:20:05.206178 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:05.206180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:05.206227 (+ 47us) server_negotiation.cc:407] Connection header received
0810 04:20:05.206370 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:05.206372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:05.206382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:05.206445 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:05.206668 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.207449 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.208981 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.209537 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.209655 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:05.209663 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:05.209838 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 04:20:05.209883 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 04:20:05.209955 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:05.206140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:52317 (local address 127.15.216.66:37567)
0810 04:20:05.206187 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:05.206189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:05.206349 (+ 160us) server_negotiation.cc:407] Connection header received
0810 04:20:05.206379 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:05.206380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:05.206393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:05.206445 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:05.206652 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.207404 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.208977 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.209537 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.209642 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:05.209652 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:05.209838 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 04:20:05.209883 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:20:05.209987 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:05.206033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:34069 (local address 127.15.216.65:60203)
0810 04:20:05.206159 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:05.206170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:05.206233 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:05.206464 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:05.206465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:05.206468 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:05.206572 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:05.206575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.207526 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.207526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:05.208847 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:05.208851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.209656 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.209656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:05.209733 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:05.209740 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 04:20:05.209802 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 04:20:05.209853 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":127,"thread_start_us":60,"threads_started":1}
I20260810 04:20:05.209993 3083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:05.206045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:37567 (local address 127.15.216.65:52317)
0810 04:20:05.206242 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:05.206250 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:05.206301 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:05.206464 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:05.206465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:05.206466 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:05.206572 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:05.206576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.207516 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.207517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:05.208847 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:05.208859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.209642 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.209643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:05.209733 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:05.209741 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 04:20:05.209802 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:20:05.209853 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":46,"thread_start_us":53,"threads_started":1}
I20260810 04:20:05.210325 3006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64e5d53a7384045b32e1f17493ee49f" candidate_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e415643d2e3a4216949a53e473893b75" is_pre_election: true
I20260810 04:20:05.210500 3006 raft_consensus.cc:2454] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a2e6487d474f61806cf3fe82dc76fa in term 0.
I20260810 04:20:05.210609 2859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64e5d53a7384045b32e1f17493ee49f" candidate_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" is_pre_election: true
I20260810 04:20:05.210788 2859 raft_consensus.cc:2454] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7a2e6487d474f61806cf3fe82dc76fa in term 0.
I20260810 04:20:05.210763 2645 leader_election.cc:302] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7a2e6487d474f61806cf3fe82dc76fa, e415643d2e3a4216949a53e473893b75; no voters:
I20260810 04:20:05.210943 3082 raft_consensus.cc:2781] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:20:05.211037 3082 raft_consensus.cc:482] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:20:05.211066 3082 raft_consensus.cc:3037] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:05.211843 3082 raft_consensus.cc:504] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:05.212000 3082 leader_election.cc:288] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [CANDIDATE]: Term 1 election: Requested vote from peers 8fde6a5c3f1547d6a7dc325b4a03c4a7 (127.15.216.66:37567), e415643d2e3a4216949a53e473893b75 (127.15.216.67:34069)
I20260810 04:20:05.212183 2859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64e5d53a7384045b32e1f17493ee49f" candidate_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7"
I20260810 04:20:05.212265 2859 raft_consensus.cc:3037] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:05.212313 3006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64e5d53a7384045b32e1f17493ee49f" candidate_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e415643d2e3a4216949a53e473893b75"
I20260810 04:20:05.212450 3006 raft_consensus.cc:3037] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:05.213120 2859 raft_consensus.cc:2454] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a2e6487d474f61806cf3fe82dc76fa in term 1.
I20260810 04:20:05.213335 2641 leader_election.cc:302] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fde6a5c3f1547d6a7dc325b4a03c4a7, c7a2e6487d474f61806cf3fe82dc76fa; no voters:
I20260810 04:20:05.213424 3006 raft_consensus.cc:2454] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7a2e6487d474f61806cf3fe82dc76fa in term 1.
I20260810 04:20:05.213443 3082 raft_consensus.cc:2781] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:05.213626 3082 raft_consensus.cc:686] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 1 LEADER]: Becoming Leader. State: Replica: c7a2e6487d474f61806cf3fe82dc76fa, State: Running, Role: LEADER
I20260810 04:20:05.213743 3082 consensus_queue.cc:237] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } }
I20260810 04:20:05.214659 2562 catalog_manager.cc:5495] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa reported cstate change: term changed from 0 to 1, leader changed from <none> to c7a2e6487d474f61806cf3fe82dc76fa (127.15.216.65). New cstate: current_term: 1 leader_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7a2e6487d474f61806cf3fe82dc76fa" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 36737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 } health_report { overall_health: UNKNOWN } } }
I20260810 04:20:05.264640 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:05.262027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.15.216.65:36737)
0810 04:20:05.262104 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:05.262106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:05.262118 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:20:05.262179 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:05.262180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:05.262202 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:05.262296 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:05.262443 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.262964 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.263824 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:05.264019 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:05.264096 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:05.264308 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:05.264409 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:05.264508 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:05.264528 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 04:20:05.264549 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:20:05.266567 2859 raft_consensus.cc:1261] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 1 FOLLOWER]: Refusing update from remote peer c7a2e6487d474f61806cf3fe82dc76fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:05.266587 3006 raft_consensus.cc:1261] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 1 FOLLOWER]: Refusing update from remote peer c7a2e6487d474f61806cf3fe82dc76fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:05.266945 3082 consensus_queue.cc:1035] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fde6a5c3f1547d6a7dc325b4a03c4a7" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:05.267105 3085 consensus_queue.cc:1035] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e415643d2e3a4216949a53e473893b75" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 34069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:05.269845 3091 mvcc.cc:205] Tried to move back new op lower bound from 7316830639169048576 to 7316830638956314624. Current Snapshot: MvccSnapshot[applied={T|T < 7316830639169048576}]
I20260810 04:20:05.269505 3088 mvcc.cc:205] Tried to move back new op lower bound from 7316830639169048576 to 7316830638956314624. Current Snapshot: MvccSnapshot[applied={T|T < 7316830639169048576}]
I20260810 04:20:05.272651 2562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:05.272756 2562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:05.273690 2562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa6b4c14d3e94aa2bef91bb7fd1f287c: Sending DeleteTablet for 3 replicas of tablet e64e5d53a7384045b32e1f17493ee49f
I20260810 04:20:05.274050 2698 tablet_service.cc:1514] Processing DeleteTablet for tablet e64e5d53a7384045b32e1f17493ee49f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:05 UTC) from {username='slave'} at 127.0.0.1:39676
I20260810 04:20:05.274114 2839 tablet_service.cc:1514] Processing DeleteTablet for tablet e64e5d53a7384045b32e1f17493ee49f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:05 UTC) from {username='slave'} at 127.0.0.1:44828
I20260810 04:20:05.274289 2980 tablet_service.cc:1514] Processing DeleteTablet for tablet e64e5d53a7384045b32e1f17493ee49f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:05 UTC) from {username='slave'} at 127.0.0.1:57926
I20260810 04:20:05.274519 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 2626
I20260810 04:20:05.274600 3101 tablet_replica.cc:330] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa: stopping tablet replica
I20260810 04:20:05.274713 3101 raft_consensus.cc:2229] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:20:05.274837 3101 raft_consensus.cc:2258] T e64e5d53a7384045b32e1f17493ee49f P c7a2e6487d474f61806cf3fe82dc76fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:05.277799 3103 tablet_replica.cc:330] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: stopping tablet replica
I20260810 04:20:05.277952 3103 raft_consensus.cc:2229] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:05.278101 3103 raft_consensus.cc:2258] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:05.278308 3102 tablet_replica.cc:330] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: stopping tablet replica
I20260810 04:20:05.278488 3102 raft_consensus.cc:2229] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:05.278652 3102 raft_consensus.cc:2258] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:05.278641 3103 ts_tablet_manager.cc:1905] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:05.280294 3103 ts_tablet_manager.cc:1918] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:05.280402 3103 log.cc:1196] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/e64e5d53a7384045b32e1f17493ee49f
I20260810 04:20:05.280745 3103 ts_tablet_manager.cc:1939] T e64e5d53a7384045b32e1f17493ee49f P e415643d2e3a4216949a53e473893b75: Deleting consensus metadata
I20260810 04:20:05.281234 2549 catalog_manager.cc:4841] TS e415643d2e3a4216949a53e473893b75 (127.15.216.67:34069): tablet e64e5d53a7384045b32e1f17493ee49f (table test-table [id=768b7677f5734514a497edb896d62d16]) successfully deleted
I20260810 04:20:05.281297 3102 ts_tablet_manager.cc:1905] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:05.283329 3102 ts_tablet_manager.cc:1918] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:05.283416 3102 log.cc:1196] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/e64e5d53a7384045b32e1f17493ee49f
I20260810 04:20:05.283680 3102 ts_tablet_manager.cc:1939] T e64e5d53a7384045b32e1f17493ee49f P 8fde6a5c3f1547d6a7dc325b4a03c4a7: Deleting consensus metadata
W20260810 04:20:05.283864 2549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:36737 (error 108)
I20260810 04:20:05.284171 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 2770
I20260810 04:20:05.284364 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:05.284231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:36737 (local address 127.0.0.1:39692)
0810 04:20:05.284285 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:05.284325 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:36737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:20:05.287107 2549 catalog_manager.cc:4584] TS c7a2e6487d474f61806cf3fe82dc76fa (127.15.216.65:36737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e64e5d53a7384045b32e1f17493ee49f: Network error: Client connection negotiation failed: client connection to 127.15.216.65:36737: connect: Connection refused (error 111)
I20260810 04:20:05.290747 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:05.290614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:37567 (local address 127.0.0.1:44832)
0810 04:20:05.290682 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:05.290714 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.66:37567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:20:05.290774 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 2911
I20260810 04:20:05.298840 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 2530
2026-08-10T04:20:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (933 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:05 dist-test-slave-g350 krb5kdc[3107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:05 dist-test-slave-g350 krb5kdc[3107](info): set up 2 sockets
Aug 10 04:20:05 dist-test-slave-g350 krb5kdc[3107](info): commencing operation
krb5kdc: starting...
W20260810 04:20:06.406793 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:06 dist-test-slave-g350 krb5kdc[3107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:06.539237 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:33701
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:33701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:06.626797 3122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:06.627048 3122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:06.627089 3122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:06.628260 3122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:06.628332 3122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:06.628356 3122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:06.628398 3122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:06.628418 3122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:06.630074 3122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:33701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:33701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:06.630342 3122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:06.630587 3122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:06.635813 3128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:06.635764 3130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:06.635764 3127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:06.636076 3122 server_base.cc:1034] running on GCE node
I20260810 04:20:06.636370 3122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:06.636693 3122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:06.637902 3122 hybrid_clock.cc:648] HybridClock initialized: now 1786335606637873 us; error 49 us; skew 500 ppm
Aug 10 04:20:06 dist-test-slave-g350 krb5kdc[3107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335606, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:06.640561 3122 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:06.640969 3122 webserver.cc:460] Webserver started at http://127.15.216.126:44951/ using document root <none> and password file <none>
I20260810 04:20:06.641156 3122 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:06.641197 3122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:06.641290 3122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:06.642398 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "4f19bce8874f423b95e8799f847acb7c"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "7afb4f2d955103e2986e5c10ca33468f"
server_key_iv: "bea2d7ba256861010e7dfc80fb1119d7"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.642722 3122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "4f19bce8874f423b95e8799f847acb7c"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "7afb4f2d955103e2986e5c10ca33468f"
server_key_iv: "bea2d7ba256861010e7dfc80fb1119d7"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.644011 3122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:20:06.644711 3138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:06.644918 3122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:06.645007 3122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "4f19bce8874f423b95e8799f847acb7c"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "7afb4f2d955103e2986e5c10ca33468f"
server_key_iv: "bea2d7ba256861010e7dfc80fb1119d7"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.645062 3122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:06.656505 3122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:06.658898 3122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:06.659022 3122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:06.663211 3122 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:33701
I20260810 04:20:06.663328 3197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:33701 every 8 connection(s)
I20260810 04:20:06.663671 3122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:06.664649 3198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:06.665287 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 3122
I20260810 04:20:06.665410 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:06.665670 16225 external_mini_cluster.cc:1428] Setting key 50d16507bf7b29c8b244763ae0196ca5
I20260810 04:20:06.667989 3198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c: Bootstrap starting.
I20260810 04:20:06.668713 3198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:06.669032 3198 log.cc:824] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:06.669801 3198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c: No bootstrap required, opened a new log
I20260810 04:20:06.671851 3198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bce8874f423b95e8799f847acb7c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33701 } }
I20260810 04:20:06.672029 3198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:06.672082 3198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f19bce8874f423b95e8799f847acb7c, State: Initialized, Role: FOLLOWER
I20260810 04:20:06.672253 3198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bce8874f423b95e8799f847acb7c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33701 } }
I20260810 04:20:06.672360 3198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:06.672407 3198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:06.672459 3198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:06.673030 3198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bce8874f423b95e8799f847acb7c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33701 } }
Aug 10 04:20:06 dist-test-slave-g350 krb5kdc[3107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:06.673161 3198 leader_election.cc:302] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f19bce8874f423b95e8799f847acb7c; no voters:
I20260810 04:20:06.673358 3198 leader_election.cc:288] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:06.673413 3203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:06.673632 3203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [term 1 LEADER]: Becoming Leader. State: Replica: 4f19bce8874f423b95e8799f847acb7c, State: Running, Role: LEADER
I20260810 04:20:06.673683 3198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:06.673738 3203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bce8874f423b95e8799f847acb7c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33701 } }
I20260810 04:20:06.674094 3203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f19bce8874f423b95e8799f847acb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bce8874f423b95e8799f847acb7c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33701 } } }
I20260810 04:20:06.674171 3203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:06.674338 3203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f19bce8874f423b95e8799f847acb7c. Latest consensus state: current_term: 1 leader_uuid: "4f19bce8874f423b95e8799f847acb7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f19bce8874f423b95e8799f847acb7c" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 33701 } } }
I20260810 04:20:06.674434 3203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:06.674587 3209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:06.675292 3209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:06.677968 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:06.666228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.15.216.126:33701)
0810 04:20:06.666485 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:06.666495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:06.666529 (+ 34us) server_negotiation.cc:407] Connection header received
0810 04:20:06.667092 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:06.667101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:06.667163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:06.667318 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:06.667978 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:06.669357 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:06.670325 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:06.670780 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:06.670925 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:06.673808 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:06.673812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:06.673812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:06.673825 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:06.676221 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:06.676658 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:06.676661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:06.676661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:06.676748 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:06.677005 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:06.677006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:06.677006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:06.677228 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:06.677338 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:06.677517 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 04:20:06.677721 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":65,"threads_started":1}
I20260810 04:20:06.678331 3209 catalog_manager.cc:1347] Generated new cluster ID: 29d7addb1a6d4b0d898afd33030adb4d
I20260810 04:20:06.678388 3209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:06.690073 3209 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:06.690884 3209 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:06.693624 3209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c: Generated new TSK 0
I20260810 04:20:06.693816 3209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:20:06.726044 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33701
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:06.819500 3224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:06.819774 3224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:06.819818 3224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:06.821053 3224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:06.821134 3224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:06.821185 3224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:20:06.822885 3224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:06.823168 3224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:06.823429 3224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:06.828923 3230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:06.828974 3233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:06.828888 3231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:06.829231 3224 server_base.cc:1034] running on GCE node
I20260810 04:20:06.829569 3224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:06.829945 3224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:06.831216 3224 hybrid_clock.cc:648] HybridClock initialized: now 1786335606831174 us; error 54 us; skew 500 ppm
Aug 10 04:20:06 dist-test-slave-g350 krb5kdc[3107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335606, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:06.833984 3224 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:20:06.834472 3224 webserver.cc:460] Webserver started at http://127.15.216.65:35779/ using document root <none> and password file <none>
I20260810 04:20:06.834717 3224 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:06.834765 3224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:06.834906 3224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:06.836024 3224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "a86f15e74efa4c0a8e955f90de7bcd92"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "5e0a64f711bf655aa255098a4c46e0f6"
server_key_iv: "73d515bf7c74818a7a8df7f6fc34bf68"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.836407 3224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "a86f15e74efa4c0a8e955f90de7bcd92"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "5e0a64f711bf655aa255098a4c46e0f6"
server_key_iv: "73d515bf7c74818a7a8df7f6fc34bf68"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.837900 3224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:20:06.838739 3243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:06.838933 3224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:06.839027 3224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "a86f15e74efa4c0a8e955f90de7bcd92"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "5e0a64f711bf655aa255098a4c46e0f6"
server_key_iv: "73d515bf7c74818a7a8df7f6fc34bf68"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.839085 3224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:06.847265 3224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:06.849830 3224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:06.849987 3224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:06.850243 3224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:06.850591 3224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:06.850627 3224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:06.850651 3224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:06.850687 3224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:06.856671 3224 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:43851
I20260810 04:20:06.856753 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:43851 every 8 connection(s)
I20260810 04:20:06.857121 3224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:20:06.863271 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 3224
I20260810 04:20:06.863379 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
Aug 10 04:20:06 dist-test-slave-g350 krb5kdc[3107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335606, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:06.863631 16225 external_mini_cluster.cc:1428] Setting key 74204edd3b954f70887f23a0666ccadc
I20260810 04:20:06.867620 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:06.858004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:47441 (local address 127.15.216.126:33701)
0810 04:20:06.858093 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:06.858095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:06.858260 (+ 165us) server_negotiation.cc:407] Connection header received
0810 04:20:06.858796 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:06.858797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:06.858807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:06.858874 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:06.859892 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:06.860392 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:06.861400 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:06.861643 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:06.861716 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:06.864276 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:06.864277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:06.864278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:06.864284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:06.865985 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:06.866579 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:06.866580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:06.866580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:06.866637 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:06.866943 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:06.866944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:06.866944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:06.867047 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:06.867100 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:06.867474 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 04:20:06.867553 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 04:20:06.867895 3365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:06.857939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:33701 (local address 127.15.216.65:47441)
0810 04:20:06.858116 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:06.858158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:06.858677 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:06.858931 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:06.858935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:06.859569 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:06.859786 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:06.859791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:06.860499 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:06.860500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:06.861304 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:06.861306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:06.861723 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:06.861724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:06.861842 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:06.862323 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:06.862332 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:06.863987 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:06.866115 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:06.866122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:06.866124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:06.866465 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:06.866716 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:06.866717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:06.866717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:06.866871 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:06.867117 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:06.867123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:06.867296 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 04:20:06.867433 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 04:20:06.867623 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260810 04:20:06.868458 3363 heartbeater.cc:347] Connected to a master server at 127.15.216.126:33701
I20260810 04:20:06.868556 3363 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:06.868762 3363 heartbeater.cc:511] Master 127.15.216.126:33701 requested a full tablet report, sending...
I20260810 04:20:06.869311 3159 ts_manager.cc:194] Registered new tserver with Master: a86f15e74efa4c0a8e955f90de7bcd92 (127.15.216.65:43851)
I20260810 04:20:06.870030 3159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:47441
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:20:06.896204 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33701
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:06.980985 3368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:06.981195 3368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:06.981220 3368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:06.982450 3368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:06.982496 3368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:06.982522 3368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:20:06.984244 3368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:06.984560 3368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:06.984810 3368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:06.990118 3376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:06.990149 3379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:06.990217 3368 server_base.cc:1034] running on GCE node
W20260810 04:20:06.990149 3374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:06.990572 3368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:06.990815 3368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:06.991990 3368 hybrid_clock.cc:648] HybridClock initialized: now 1786335606991971 us; error 30 us; skew 500 ppm
Aug 10 04:20:06 dist-test-slave-g350 krb5kdc[3107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335606, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:06.994678 3368 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:20:06.995172 3368 webserver.cc:460] Webserver started at http://127.15.216.66:40619/ using document root <none> and password file <none>
I20260810 04:20:06.995405 3368 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:06.995478 3368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:06.995644 3368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:06.996737 3368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "54d1ddfa7e6b40e1a7e26f92c5195072"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "5409f3cfcb9d492bae53ba700bbb5eff"
server_key_iv: "082b6ca5ae88c7317266b498aed391ea"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.997135 3368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "54d1ddfa7e6b40e1a7e26f92c5195072"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "5409f3cfcb9d492bae53ba700bbb5eff"
server_key_iv: "082b6ca5ae88c7317266b498aed391ea"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.998603 3368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:06.999416 3386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:06.999608 3368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:06.999720 3368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "54d1ddfa7e6b40e1a7e26f92c5195072"
format_stamp: "Formatted at 2026-08-10 04:20:06 on dist-test-slave-g350"
server_key: "5409f3cfcb9d492bae53ba700bbb5eff"
server_key_iv: "082b6ca5ae88c7317266b498aed391ea"
server_key_version: "encryptionkey@0"
I20260810 04:20:06.999799 3368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:07.009361 3368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:07.011833 3368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:07.012005 3368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:07.012251 3368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:07.012611 3368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:07.012686 3368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.012759 3368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:07.012804 3368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.019939 3368 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:45441
I20260810 04:20:07.019986 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:45441 every 8 connection(s)
I20260810 04:20:07.020341 3368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:20:07.022070 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 3368
I20260810 04:20:07.022235 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:20:07.022454 16225 external_mini_cluster.cc:1428] Setting key 7e23d9e5e1b763018479905a219174d5
Aug 10 04:20:07 dist-test-slave-g350 krb5kdc[3107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335606, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:07.031906 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.021166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:33659 (local address 127.15.216.126:33701)
0810 04:20:07.021216 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.021218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.021472 (+ 254us) server_negotiation.cc:407] Connection header received
0810 04:20:07.022606 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.022608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.022622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.022685 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:07.023435 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.023926 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.025151 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.025475 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.025520 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.028637 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:07.028638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:07.028638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:07.028645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:07.030186 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:07.030816 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:07.030818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:07.030818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:07.030888 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:07.031228 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:07.031228 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:07.031228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:07.031331 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:07.031391 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.031761 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.031826 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:20:07.032274 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.021134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:33701 (local address 127.15.216.66:33659)
0810 04:20:07.021335 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.021373 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.022459 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.022753 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.022757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.023137 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:07.023331 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.023336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.024031 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.024032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.024830 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.024832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.025643 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.025644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.025792 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.026367 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:07.026376 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:07.028531 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:07.030280 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:07.030289 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:07.030292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:07.030674 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:07.030969 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:07.030970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:07.030970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:07.031134 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:07.031414 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:07.031423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:07.031606 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.031744 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.031944 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":131,"threads_started":1}
I20260810 04:20:07.032897 3508 heartbeater.cc:347] Connected to a master server at 127.15.216.126:33701
I20260810 04:20:07.033013 3508 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
I20260810 04:20:07.033224 3508 heartbeater.cc:511] Master 127.15.216.126:33701 requested a full tablet report, sending...
I20260810 04:20:07.033814 3159 ts_manager.cc:194] Registered new tserver with Master: 54d1ddfa7e6b40e1a7e26f92c5195072 (127.15.216.66:45441)
I20260810 04:20:07.034257 3159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:33659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:20:07.054116 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33701
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:07.139703 3514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:07.139904 3514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:07.139928 3514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:07.141021 3514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:07.141060 3514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:07.141086 3514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:20:07.142720 3514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:33701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:07.142917 3514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:07.143092 3514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:07.148648 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.148627 3519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.148650 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:07.148890 3514 server_base.cc:1034] running on GCE node
I20260810 04:20:07.149168 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:07.149415 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:07.150564 3514 hybrid_clock.cc:648] HybridClock initialized: now 1786335607150547 us; error 34 us; skew 500 ppm
Aug 10 04:20:07 dist-test-slave-g350 krb5kdc[3107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335607, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:07.158245 3514 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:20:07.158839 3514 webserver.cc:460] Webserver started at http://127.15.216.67:41567/ using document root <none> and password file <none>
I20260810 04:20:07.159108 3514 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:07.159179 3514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:07.159328 3514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:07.160812 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "ff45f81b27cf4fa98159913dccee2441"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "edc009a41472036f1d6a46dc04818b80"
server_key_iv: "b0e35d3099664f82c2acfddc8ed50a54"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.161204 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "ff45f81b27cf4fa98159913dccee2441"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "edc009a41472036f1d6a46dc04818b80"
server_key_iv: "b0e35d3099664f82c2acfddc8ed50a54"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.162685 3514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260810 04:20:07.163573 3529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.163777 3514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:07.163904 3514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "ff45f81b27cf4fa98159913dccee2441"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "edc009a41472036f1d6a46dc04818b80"
server_key_iv: "b0e35d3099664f82c2acfddc8ed50a54"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.163995 3514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:07.191777 3514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:07.194339 3514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:07.194502 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:07.194761 3514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:07.195112 3514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:07.195148 3514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.195194 3514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:07.195221 3514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.201565 3514 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:37513
I20260810 04:20:07.201685 3653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:37513 every 8 connection(s)
I20260810 04:20:07.202051 3514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
Aug 10 04:20:07 dist-test-slave-g350 krb5kdc[3107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335607, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:07.210654 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 3514
I20260810 04:20:07.210798 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:20:07.211009 16225 external_mini_cluster.cc:1428] Setting key c7ea238e3e58294537406cf62eaba1aa
I20260810 04:20:07.212286 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.202825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:44833 (local address 127.15.216.126:33701)
0810 04:20:07.202883 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.202885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.203163 (+ 278us) server_negotiation.cc:407] Connection header received
0810 04:20:07.203754 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.203755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.203766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.203870 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:07.204887 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.205391 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.206538 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.206721 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.206781 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.208999 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:07.209001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:07.209001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:07.209008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:07.210637 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:07.211193 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:07.211195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:07.211195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:07.211272 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:07.211573 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:07.211574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:07.211574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:07.211698 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:07.211802 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.212096 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.212181 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:20:07.212486 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.202821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:33701 (local address 127.15.216.67:44833)
0810 04:20:07.203009 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.203067 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.203653 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.203947 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.203951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.204564 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:07.204775 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.204780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.205504 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.205505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.206453 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.206456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.206807 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.206808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.206917 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.207463 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:07.207472 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:07.208871 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:07.210738 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:07.210748 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:07.210750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:07.211092 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:07.211356 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:07.211357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:07.211357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:07.211490 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:07.211794 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:07.211802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:07.211954 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.212073 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.212245 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260810 04:20:07.213040 3654 heartbeater.cc:347] Connected to a master server at 127.15.216.126:33701
I20260810 04:20:07.213129 3654 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:07.213392 3654 heartbeater.cc:511] Master 127.15.216.126:33701 requested a full tablet report, sending...
I20260810 04:20:07.213891 3159 ts_manager.cc:194] Registered new tserver with Master: ff45f81b27cf4fa98159913dccee2441 (127.15.216.67:37513)
I20260810 04:20:07.214267 3159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:44833
I20260810 04:20:07.223014 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:20:07.230868 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.224457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.15.216.126:33701)
0810 04:20:07.224534 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.224536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.224553 (+ 17us) server_negotiation.cc:407] Connection header received
0810 04:20:07.224608 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.224609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.224618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.224678 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:07.225237 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.225858 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.226585 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.226783 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.226838 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.228132 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:07.228133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:07.228134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:07.228140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:07.229756 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:07.230163 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:07.230165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:07.230165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:07.230224 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:07.230444 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:07.230445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:07.230445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:07.230574 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:07.230658 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.230752 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.230803 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:20:07.232591 3159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:20:07.234319 3159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:20:07.240899 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.236587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:43851 (local address 127.0.0.1:32922)
0810 04:20:07.236803 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.236845 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.236912 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.237382 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.237382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.237384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:07.237416 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.237417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.238765 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.238766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.239932 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.239934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.240586 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.240587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.240681 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.240695 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.240770 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.240824 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":7,"thread_start_us":56,"threads_started":1}
I20260810 04:20:07.240957 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.236685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.15.216.65:43851)
0810 04:20:07.237221 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.237225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.237234 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:07.237282 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.237284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.237300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.237391 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:07.237477 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.238656 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.240024 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.240495 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.240586 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.240649 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.240822 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.240871 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":101,"threads_started":1}
I20260810 04:20:07.241832 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.236636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:37513 (local address 127.0.0.1:43782)
0810 04:20:07.236879 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.236890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.236945 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.237317 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.237320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.237323 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:07.237408 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.237410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.238600 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.238601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.239930 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.239934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.241559 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.241560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.241648 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.241652 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.241713 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.241753 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260810 04:20:07.241881 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.236941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.15.216.67:37513)
0810 04:20:07.237111 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.237116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.237126 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:20:07.237172 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.237174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.237191 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.237713 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:07.237789 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.238491 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.240689 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.241464 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.241537 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.241610 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.241741 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.241781 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260810 04:20:07.241889 3292 tablet_service.cc:1467] Processing CreateTablet for tablet 70ea6cab8745478083d9ccd861e4898e (DEFAULT_TABLE table=test-table [id=2154afac7b84424d9c8a9745d124b3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:07.242268 3292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ea6cab8745478083d9ccd861e4898e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:07.242312 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.236799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:45441 (local address 127.0.0.1:52834)
0810 04:20:07.237136 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.237147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.237203 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.237489 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.237489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.237491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:07.237534 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.237535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.240029 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.240030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.240716 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.240718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.242043 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.242044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.242115 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.242120 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.242185 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.242233 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":68,"threads_started":1}
I20260810 04:20:07.242575 3582 tablet_service.cc:1467] Processing CreateTablet for tablet 70ea6cab8745478083d9ccd861e4898e (DEFAULT_TABLE table=test-table [id=2154afac7b84424d9c8a9745d124b3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:07.242832 3582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ea6cab8745478083d9ccd861e4898e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:07.244814 3671 tablet_bootstrap.cc:492] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92: Bootstrap starting.
I20260810 04:20:07.245325 3672 tablet_bootstrap.cc:492] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Bootstrap starting.
I20260810 04:20:07.245541 3671 tablet_bootstrap.cc:654] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:07.245651 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.236927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.15.216.66:45441)
0810 04:20:07.237137 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.237141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.237196 (+ 55us) server_negotiation.cc:407] Connection header received
0810 04:20:07.237311 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.237313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.237330 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.237503 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:07.237579 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.238656 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.240846 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.241955 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.242029 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.242090 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.245486 (+ 3396us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.245543 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260810 04:20:07.246074 3672 tablet_bootstrap.cc:654] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:07.246203 3671 log.cc:824] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:07.246380 3438 tablet_service.cc:1467] Processing CreateTablet for tablet 70ea6cab8745478083d9ccd861e4898e (DEFAULT_TABLE table=test-table [id=2154afac7b84424d9c8a9745d124b3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:07.246634 3672 log.cc:824] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:07.246731 3438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ea6cab8745478083d9ccd861e4898e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:07.247095 3671 tablet_bootstrap.cc:492] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92: No bootstrap required, opened a new log
I20260810 04:20:07.247172 3671 ts_tablet_manager.cc:1397] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:20:07.247380 3672 tablet_bootstrap.cc:492] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: No bootstrap required, opened a new log
I20260810 04:20:07.247449 3672 ts_tablet_manager.cc:1397] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:20:07.249284 3671 raft_consensus.cc:348] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.249478 3671 raft_consensus.cc:374] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:07.249519 3671 raft_consensus.cc:729] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86f15e74efa4c0a8e955f90de7bcd92, State: Initialized, Role: FOLLOWER
I20260810 04:20:07.249706 3671 consensus_queue.cc:260] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.249553 3672 raft_consensus.cc:348] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.249876 3672 raft_consensus.cc:374] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:07.249938 3672 raft_consensus.cc:729] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff45f81b27cf4fa98159913dccee2441, State: Initialized, Role: FOLLOWER
I20260810 04:20:07.250056 3671 ts_tablet_manager.cc:1428] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:20:07.250058 3672 consensus_queue.cc:260] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.250185 3363 heartbeater.cc:503] Master 127.15.216.126:33701 was elected leader, sending a full tablet report...
I20260810 04:20:07.250337 3654 heartbeater.cc:503] Master 127.15.216.126:33701 was elected leader, sending a full tablet report...
I20260810 04:20:07.250344 3672 ts_tablet_manager.cc:1428] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:20:07.250471 3675 tablet_bootstrap.cc:492] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Bootstrap starting.
I20260810 04:20:07.251149 3675 tablet_bootstrap.cc:654] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:07.251721 3675 log.cc:824] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:07.252470 3675 tablet_bootstrap.cc:492] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: No bootstrap required, opened a new log
I20260810 04:20:07.252523 3675 ts_tablet_manager.cc:1397] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:20:07.253986 3675 raft_consensus.cc:348] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.254112 3675 raft_consensus.cc:374] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:07.254133 3675 raft_consensus.cc:729] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d1ddfa7e6b40e1a7e26f92c5195072, State: Initialized, Role: FOLLOWER
I20260810 04:20:07.254290 3675 consensus_queue.cc:260] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.254556 3675 ts_tablet_manager.cc:1428] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:20:07.254595 3508 heartbeater.cc:503] Master 127.15.216.126:33701 was elected leader, sending a full tablet report...
W20260810 04:20:07.270915 3509 tablet.cc:2369] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:07.358080 3364 tablet.cc:2369] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:20:07.395362 3676 raft_consensus.cc:482] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:07.395498 3676 raft_consensus.cc:504] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.395932 3676 leader_election.cc:288] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54d1ddfa7e6b40e1a7e26f92c5195072 (127.15.216.66:45441), ff45f81b27cf4fa98159913dccee2441 (127.15.216.67:37513)
I20260810 04:20:07.400036 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.396278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:45403 (local address 127.15.216.66:45441)
0810 04:20:07.396321 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.396322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.396351 (+ 29us) server_negotiation.cc:407] Connection header received
0810 04:20:07.396426 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.396427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.396440 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.396517 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:07.396767 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.397511 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.399064 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.399641 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.399720 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.399729 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.399926 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.399966 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:20:07.400036 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.396240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:43775 (local address 127.15.216.67:37513)
0810 04:20:07.396291 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.396294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.396349 (+ 55us) server_negotiation.cc:407] Connection header received
0810 04:20:07.396438 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.396448 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.396457 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.396517 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:07.396764 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.397507 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.399068 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.399641 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.399720 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.399729 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.399928 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.399967 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 04:20:07.400060 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.396130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:45441 (local address 127.15.216.65:45403)
0810 04:20:07.396285 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.396296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.396362 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.396541 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.396542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.396544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:07.396657 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.396660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.397627 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.397628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.398964 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.398967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.399737 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.399738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.399831 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.399838 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.399901 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.399953 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":85,"thread_start_us":80,"threads_started":1}
I20260810 04:20:07.400064 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.396128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:37513 (local address 127.15.216.65:43775)
0810 04:20:07.396284 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.396296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.396362 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.396541 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.396542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.396545 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:07.396685 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.396686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.397625 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.397626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.398964 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.398967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.399738 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.399739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.399831 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.399840 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.399901 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.399954 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":59,"thread_start_us":84,"threads_started":1}
I20260810 04:20:07.400509 3455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ea6cab8745478083d9ccd861e4898e" candidate_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" is_pre_election: true
I20260810 04:20:07.400683 3455 raft_consensus.cc:2454] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86f15e74efa4c0a8e955f90de7bcd92 in term 0.
I20260810 04:20:07.400681 3602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ea6cab8745478083d9ccd861e4898e" candidate_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45f81b27cf4fa98159913dccee2441" is_pre_election: true
I20260810 04:20:07.400867 3602 raft_consensus.cc:2454] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86f15e74efa4c0a8e955f90de7bcd92 in term 0.
I20260810 04:20:07.400981 3244 leader_election.cc:302] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54d1ddfa7e6b40e1a7e26f92c5195072, a86f15e74efa4c0a8e955f90de7bcd92; no voters:
I20260810 04:20:07.401132 3676 raft_consensus.cc:2781] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:20:07.401216 3676 raft_consensus.cc:482] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:20:07.401234 3676 raft_consensus.cc:3037] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:07.402114 3676 raft_consensus.cc:504] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.402284 3676 leader_election.cc:288] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [CANDIDATE]: Term 1 election: Requested vote from peers 54d1ddfa7e6b40e1a7e26f92c5195072 (127.15.216.66:45441), ff45f81b27cf4fa98159913dccee2441 (127.15.216.67:37513)
I20260810 04:20:07.402451 3602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ea6cab8745478083d9ccd861e4898e" candidate_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff45f81b27cf4fa98159913dccee2441"
I20260810 04:20:07.402448 3455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ea6cab8745478083d9ccd861e4898e" candidate_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072"
I20260810 04:20:07.402523 3602 raft_consensus.cc:3037] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:07.402549 3455 raft_consensus.cc:3037] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:07.403334 3602 raft_consensus.cc:2454] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86f15e74efa4c0a8e955f90de7bcd92 in term 1.
I20260810 04:20:07.403373 3455 raft_consensus.cc:2454] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86f15e74efa4c0a8e955f90de7bcd92 in term 1.
I20260810 04:20:07.403512 3244 leader_election.cc:302] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54d1ddfa7e6b40e1a7e26f92c5195072, a86f15e74efa4c0a8e955f90de7bcd92; no voters:
I20260810 04:20:07.403604 3676 raft_consensus.cc:2781] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:07.403754 3676 raft_consensus.cc:686] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [term 1 LEADER]: Becoming Leader. State: Replica: a86f15e74efa4c0a8e955f90de7bcd92, State: Running, Role: LEADER
I20260810 04:20:07.403874 3676 consensus_queue.cc:237] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } }
I20260810 04:20:07.404704 3158 catalog_manager.cc:5495] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 reported cstate change: term changed from 0 to 1, leader changed from <none> to a86f15e74efa4c0a8e955f90de7bcd92 (127.15.216.65). New cstate: current_term: 1 leader_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a86f15e74efa4c0a8e955f90de7bcd92" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 43851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 } health_report { overall_health: UNKNOWN } } }
I20260810 04:20:07.426961 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.424458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.15.216.65:43851)
0810 04:20:07.424527 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.424529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.424537 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:07.424582 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.424583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.424600 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.424682 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:07.424847 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.425339 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.426168 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.426367 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.426458 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.426601 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:07.426695 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:07.426814 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.426865 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.426875 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 04:20:07.428995 3602 raft_consensus.cc:1261] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 1 FOLLOWER]: Refusing update from remote peer a86f15e74efa4c0a8e955f90de7bcd92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:07.428995 3455 raft_consensus.cc:1261] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 1 FOLLOWER]: Refusing update from remote peer a86f15e74efa4c0a8e955f90de7bcd92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:07.429371 3676 consensus_queue.cc:1035] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d1ddfa7e6b40e1a7e26f92c5195072" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:07.429530 3682 consensus_queue.cc:1035] T 70ea6cab8745478083d9ccd861e4898e P a86f15e74efa4c0a8e955f90de7bcd92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff45f81b27cf4fa98159913dccee2441" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 37513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:07.433019 3687 mvcc.cc:205] Tried to move back new op lower bound from 7316830648025931776 to 7316830647926874112. Current Snapshot: MvccSnapshot[applied={T|T < 7316830648025931776 or (T in {7316830648025931776})}]
I20260810 04:20:07.435336 3159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:07.435423 3159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:07.436239 3159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f19bce8874f423b95e8799f847acb7c: Sending DeleteTablet for 3 replicas of tablet 70ea6cab8745478083d9ccd861e4898e
I20260810 04:20:07.436615 3582 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ea6cab8745478083d9ccd861e4898e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:07 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43782
I20260810 04:20:07.436630 3438 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ea6cab8745478083d9ccd861e4898e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:07 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:52834
I20260810 04:20:07.436679 3292 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ea6cab8745478083d9ccd861e4898e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:07 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:32922
I20260810 04:20:07.436836 3697 tablet_replica.cc:330] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: stopping tablet replica
I20260810 04:20:07.436945 3697 raft_consensus.cc:2229] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:07.437049 3697 raft_consensus.cc:2258] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:07.437091 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 3224
I20260810 04:20:07.438653 3697 ts_tablet_manager.cc:1905] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:07.440101 3698 tablet_replica.cc:330] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: stopping tablet replica
I20260810 04:20:07.440200 3698 raft_consensus.cc:2229] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:07.440301 3698 raft_consensus.cc:2258] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:07.440815 3697 ts_tablet_manager.cc:1918] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:07.440902 3697 log.cc:1196] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/70ea6cab8745478083d9ccd861e4898e
I20260810 04:20:07.441252 3697 ts_tablet_manager.cc:1939] T 70ea6cab8745478083d9ccd861e4898e P ff45f81b27cf4fa98159913dccee2441: Deleting consensus metadata
I20260810 04:20:07.445330 3143 catalog_manager.cc:4841] TS ff45f81b27cf4fa98159913dccee2441 (127.15.216.67:37513): tablet 70ea6cab8745478083d9ccd861e4898e (table test-table [id=2154afac7b84424d9c8a9745d124b3dd]) successfully deleted
I20260810 04:20:07.445473 3698 ts_tablet_manager.cc:1905] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:07.447403 3698 ts_tablet_manager.cc:1918] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:07.447506 3698 log.cc:1196] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/70ea6cab8745478083d9ccd861e4898e
I20260810 04:20:07.447858 3698 ts_tablet_manager.cc:1939] T 70ea6cab8745478083d9ccd861e4898e P 54d1ddfa7e6b40e1a7e26f92c5195072: Deleting consensus metadata
I20260810 04:20:07.448400 3139 catalog_manager.cc:4841] TS 54d1ddfa7e6b40e1a7e26f92c5195072 (127.15.216.66:45441): tablet 70ea6cab8745478083d9ccd861e4898e (table test-table [id=2154afac7b84424d9c8a9745d124b3dd]) successfully deleted
W20260810 04:20:07.450917 3139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:43851 (error 108)
I20260810 04:20:07.451160 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 3368
I20260810 04:20:07.452330 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.452153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:43851 (local address 127.0.0.1:32942)
0810 04:20:07.452232 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.452284 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:43851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 04:20:07.452464 3139 catalog_manager.cc:4584] TS a86f15e74efa4c0a8e955f90de7bcd92 (127.15.216.65:43851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70ea6cab8745478083d9ccd861e4898e: Network error: Client connection negotiation failed: client connection to 127.15.216.65:43851: connect: Connection refused (error 111)
I20260810 04:20:07.458616 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 3514
I20260810 04:20:07.465243 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 3122
2026-08-10T04:20:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2167 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T04:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:07Z Disabled control of system clock
I20260810 04:20:07.496296 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44699
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:44699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:20:07.581902 3704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:07.582116 3704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:07.582139 3704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:07.583218 3704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:07.583252 3704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:07.583264 3704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:07.583277 3704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:07.583289 3704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:07.584568 3704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:44699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:07.584760 3704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:07.584997 3704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:07.590135 3713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.590144 3710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.590240 3711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:07.590307 3704 server_base.cc:1034] running on GCE node
I20260810 04:20:07.590647 3704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:07.590947 3704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:07.592136 3704 hybrid_clock.cc:648] HybridClock initialized: now 1786335607592121 us; error 46 us; skew 500 ppm
I20260810 04:20:07.593637 3704 webserver.cc:460] Webserver started at http://127.15.216.126:37051/ using document root <none> and password file <none>
I20260810 04:20:07.593921 3704 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:07.593998 3704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:07.594152 3704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:07.595356 3704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "9562cb40ce3343149a70042d0802f736"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "02295aa19b7dd1bd627cb0c3c675c634"
server_key_iv: "53b89575a3d2db19bc39e2d8b9d32a09"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.595808 3704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "9562cb40ce3343149a70042d0802f736"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "02295aa19b7dd1bd627cb0c3c675c634"
server_key_iv: "53b89575a3d2db19bc39e2d8b9d32a09"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.597244 3704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:07.598219 3720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.598430 3704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:20:07.598544 3704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "9562cb40ce3343149a70042d0802f736"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "02295aa19b7dd1bd627cb0c3c675c634"
server_key_iv: "53b89575a3d2db19bc39e2d8b9d32a09"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.598630 3704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:07.611620 3704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:07.611935 3704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:07.612092 3704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:07.616529 3704 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:44699
I20260810 04:20:07.616643 3782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:44699 every 8 connection(s)
I20260810 04:20:07.616947 3704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:07.618031 3783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:07.620772 3783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736: Bootstrap starting.
I20260810 04:20:07.621160 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 3704
I20260810 04:20:07.621280 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:07.621450 3783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:07.621523 16225 external_mini_cluster.cc:1428] Setting key 2803708bb157fb9748569ae9ec5fec1e
I20260810 04:20:07.621788 3783 log.cc:824] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:07.622566 3783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736: No bootstrap required, opened a new log
I20260810 04:20:07.624758 3783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562cb40ce3343149a70042d0802f736" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44699 } }
I20260810 04:20:07.624924 3783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:07.624954 3783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9562cb40ce3343149a70042d0802f736, State: Initialized, Role: FOLLOWER
I20260810 04:20:07.625068 3783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562cb40ce3343149a70042d0802f736" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44699 } }
I20260810 04:20:07.625144 3783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:07.625196 3783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:07.625286 3783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:07.626001 3783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562cb40ce3343149a70042d0802f736" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44699 } }
I20260810 04:20:07.626147 3783 leader_election.cc:302] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9562cb40ce3343149a70042d0802f736; no voters:
I20260810 04:20:07.626365 3783 leader_election.cc:288] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:07.626432 3789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:07.626592 3789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [term 1 LEADER]: Becoming Leader. State: Replica: 9562cb40ce3343149a70042d0802f736, State: Running, Role: LEADER
I20260810 04:20:07.626673 3783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:07.626683 3789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562cb40ce3343149a70042d0802f736" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44699 } }
I20260810 04:20:07.627097 3789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9562cb40ce3343149a70042d0802f736. Latest consensus state: current_term: 1 leader_uuid: "9562cb40ce3343149a70042d0802f736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562cb40ce3343149a70042d0802f736" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44699 } } }
I20260810 04:20:07.627192 3789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:07.627108 3790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9562cb40ce3343149a70042d0802f736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9562cb40ce3343149a70042d0802f736" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44699 } } }
I20260810 04:20:07.627386 3790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:07.627427 3795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:07.627599 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.622080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.15.216.126:44699)
0810 04:20:07.622294 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.622302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.622340 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:20:07.622804 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.622811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.622875 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.623008 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:07.623233 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.624080 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.625190 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.625770 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.625867 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.626807 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:07.626811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:07.626811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:07.627034 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:07.627075 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:07.627092 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:07.627136 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.627295 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.627387 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260810 04:20:07.627941 3795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:07.630784 3795 catalog_manager.cc:1347] Generated new cluster ID: d4681946b877405289bc158ffc5b0183
I20260810 04:20:07.630844 3795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:07.640365 3795 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:07.640877 3795 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:07.655893 3795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736: Generated new TSK 0
I20260810 04:20:07.656157 3795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:07.661485 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:44699
--builtin_ntp_servers=127.15.216.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:20:07.749962 3812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:07.750209 3812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:07.750247 3812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:07.751396 3812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:07.751474 3812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:07.751518 3812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:20:07.752874 3812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:44699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:07.753152 3812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:07.753669 3812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:07.759011 3822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.759152 3819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.759361 3820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:07.759475 3812 server_base.cc:1034] running on GCE node
I20260810 04:20:07.759666 3812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:07.759970 3812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:07.761152 3812 hybrid_clock.cc:648] HybridClock initialized: now 1786335607761126 us; error 39 us; skew 500 ppm
I20260810 04:20:07.762616 3812 webserver.cc:460] Webserver started at http://127.15.216.65:40143/ using document root <none> and password file <none>
I20260810 04:20:07.762889 3812 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:07.762948 3812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:07.763100 3812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:07.764249 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "b815f84640eb406aa6b8f05292845d7b"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "0ea6952f2eab8a4f08e3743b2e7b95db"
server_key_iv: "ff96900a4fe8f105e8ffb83e3df6eac3"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.764634 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "b815f84640eb406aa6b8f05292845d7b"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "0ea6952f2eab8a4f08e3743b2e7b95db"
server_key_iv: "ff96900a4fe8f105e8ffb83e3df6eac3"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.766109 3812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:07.766983 3830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.767171 3812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:07.767325 3812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "b815f84640eb406aa6b8f05292845d7b"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "0ea6952f2eab8a4f08e3743b2e7b95db"
server_key_iv: "ff96900a4fe8f105e8ffb83e3df6eac3"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.767416 3812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:07.778604 3812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:07.778936 3812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:07.779103 3812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:07.779402 3812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:07.779739 3812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:07.779798 3812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.779834 3812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:07.779881 3812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.785960 3812 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:33329
I20260810 04:20:07.786003 3967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:33329 every 8 connection(s)
I20260810 04:20:07.786516 3812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:20:07.788070 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 3812
I20260810 04:20:07.788189 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:20:07.788424 16225 external_mini_cluster.cc:1428] Setting key 248cbf050481a06522c95e110451bff1
I20260810 04:20:07.789376 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:44699
--builtin_ntp_servers=127.15.216.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:20:07.792291 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.787298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:50427 (local address 127.15.216.126:44699)
0810 04:20:07.787383 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.787386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.787609 (+ 223us) server_negotiation.cc:407] Connection header received
0810 04:20:07.788280 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.788281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.788294 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.788348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:07.789143 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.789811 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.790797 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.791016 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.791105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.791982 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:07.791984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:07.791984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:07.791989 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:07.792003 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:07.792010 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:07.792079 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.792176 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.792210 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 04:20:07.792572 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.787256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:44699 (local address 127.15.216.65:50427)
0810 04:20:07.787475 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.787508 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.788166 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.788462 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.788467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.788479 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:07.789038 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.789043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.789917 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.789918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.790700 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.790703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.791116 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.791117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.791247 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.791824 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:07.791825 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:07.791852 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:07.791875 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:07.792080 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:07.792087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:07.792091 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.792179 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.792283 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260810 04:20:07.793094 3968 heartbeater.cc:347] Connected to a master server at 127.15.216.126:44699
I20260810 04:20:07.793185 3968 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:07.793386 3968 heartbeater.cc:511] Master 127.15.216.126:44699 requested a full tablet report, sending...
I20260810 04:20:07.793910 3740 ts_manager.cc:194] Registered new tserver with Master: b815f84640eb406aa6b8f05292845d7b (127.15.216.65:33329)
I20260810 04:20:07.794525 3740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.65:50427
W20260810 04:20:07.874989 3972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:07.875238 3972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:07.875278 3972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:07.876472 3972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:07.876546 3972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:07.876590 3972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:20:07.878001 3972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:44699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:07.878295 3972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:07.878612 3972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:07.884505 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.884534 3980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:07.884505 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:07.884904 3972 server_base.cc:1034] running on GCE node
I20260810 04:20:07.885156 3972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:07.885428 3972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:07.886606 3972 hybrid_clock.cc:648] HybridClock initialized: now 1786335607886582 us; error 65 us; skew 500 ppm
I20260810 04:20:07.888031 3972 webserver.cc:460] Webserver started at http://127.15.216.66:46209/ using document root <none> and password file <none>
I20260810 04:20:07.888365 3972 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:07.888451 3972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:07.888614 3972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:07.889848 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "2ec64f39139748de80d2c6b0f68b6da6"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "785bbe167016821cc3ea5f4fb83bf130"
server_key_iv: "e70b7310199eb85d1c326c7f3060fdb0"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.890228 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "2ec64f39139748de80d2c6b0f68b6da6"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "785bbe167016821cc3ea5f4fb83bf130"
server_key_iv: "e70b7310199eb85d1c326c7f3060fdb0"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.891598 3972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:20:07.892413 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.892596 3972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:07.892711 3972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "2ec64f39139748de80d2c6b0f68b6da6"
format_stamp: "Formatted at 2026-08-10 04:20:07 on dist-test-slave-g350"
server_key: "785bbe167016821cc3ea5f4fb83bf130"
server_key_iv: "e70b7310199eb85d1c326c7f3060fdb0"
server_key_version: "encryptionkey@0"
I20260810 04:20:07.892791 3972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:07.913086 3972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:07.913461 3972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:07.913689 3972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:07.913959 3972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:07.914331 3972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:07.914390 3972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.914433 3972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:07.914462 3972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:07.920598 3972 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:41303
I20260810 04:20:07.920658 4112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:41303 every 8 connection(s)
I20260810 04:20:07.921051 3972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:20:07.925843 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 3972
I20260810 04:20:07.925963 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:20:07.926214 16225 external_mini_cluster.cc:1428] Setting key 5271943c5a3ca836e9c075659211db1a
I20260810 04:20:07.926437 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.921898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:38545 (local address 127.15.216.126:44699)
0810 04:20:07.921943 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:07.921944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:07.922189 (+ 245us) server_negotiation.cc:407] Connection header received
0810 04:20:07.922662 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:07.922664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:07.922673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:07.922734 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:07.923358 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.923838 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.924838 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.925023 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.925072 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.926061 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:07.926063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:07.926063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:07.926068 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:07.926083 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:07.926088 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:07.926136 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:07.926289 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 04:20:07.926318 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:20:07.926659 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:07.921855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:44699 (local address 127.15.216.66:38545)
0810 04:20:07.922057 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:07.922098 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:07.922581 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:07.922785 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:07.922788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:07.922796 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:07.923289 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.923292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.923930 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.923931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.924757 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:07.924759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:07.925107 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:07.925108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:07.925214 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:07.925834 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:07.925835 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:07.925873 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:07.925903 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:07.926143 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:07.926151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:07.926154 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:07.926261 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 04:20:07.926422 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260810 04:20:07.927187 4113 heartbeater.cc:347] Connected to a master server at 127.15.216.126:44699
I20260810 04:20:07.927318 4113 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:07.927343 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:44699
--builtin_ntp_servers=127.15.216.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:20:07.927521 4113 heartbeater.cc:511] Master 127.15.216.126:44699 requested a full tablet report, sending...
I20260810 04:20:07.927937 3740 ts_manager.cc:194] Registered new tserver with Master: 2ec64f39139748de80d2c6b0f68b6da6 (127.15.216.66:41303)
I20260810 04:20:07.928375 3740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.66:38545
W20260810 04:20:08.014554 4116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:08.014758 4116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:08.014780 4116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:08.015995 4116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:08.016038 4116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:08.016065 4116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:20:08.017446 4116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:44699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:08.017728 4116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:08.018038 4116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:08.023321 4121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:08.023341 4124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:08.023530 4122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:08.023646 4116 server_base.cc:1034] running on GCE node
I20260810 04:20:08.023911 4116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:08.024161 4116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:08.025303 4116 hybrid_clock.cc:648] HybridClock initialized: now 1786335608025300 us; error 57 us; skew 500 ppm
I20260810 04:20:08.026782 4116 webserver.cc:460] Webserver started at http://127.15.216.67:42067/ using document root <none> and password file <none>
I20260810 04:20:08.027042 4116 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:08.027087 4116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:08.027258 4116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:08.028437 4116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "034faa4b18bb4846a657553c523a983b"
format_stamp: "Formatted at 2026-08-10 04:20:08 on dist-test-slave-g350"
server_key: "fd5c0956328f57cc385f58144acd7e78"
server_key_iv: "803f16d73bb2719db624083ac5972f62"
server_key_version: "encryptionkey@0"
I20260810 04:20:08.028869 4116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "034faa4b18bb4846a657553c523a983b"
format_stamp: "Formatted at 2026-08-10 04:20:08 on dist-test-slave-g350"
server_key: "fd5c0956328f57cc385f58144acd7e78"
server_key_iv: "803f16d73bb2719db624083ac5972f62"
server_key_version: "encryptionkey@0"
I20260810 04:20:08.030372 4116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:08.031138 4130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:08.031425 4116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:08.031524 4116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "034faa4b18bb4846a657553c523a983b"
format_stamp: "Formatted at 2026-08-10 04:20:08 on dist-test-slave-g350"
server_key: "fd5c0956328f57cc385f58144acd7e78"
server_key_iv: "803f16d73bb2719db624083ac5972f62"
server_key_version: "encryptionkey@0"
I20260810 04:20:08.031580 4116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:08.051452 4116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:08.051734 4116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:08.051839 4116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:08.052209 4116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:08.052604 4116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:08.052640 4116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:08.052663 4116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:08.052676 4116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:08.058954 4116 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:35563
I20260810 04:20:08.059065 4255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:35563 every 8 connection(s)
I20260810 04:20:08.059401 4116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:20:08.063393 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 4116
I20260810 04:20:08.063519 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:20:08.063721 16225 external_mini_cluster.cc:1428] Setting key d776237c18a57de61275723e60e75452
I20260810 04:20:08.065481 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.060266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:56285 (local address 127.15.216.126:44699)
0810 04:20:08.060392 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.060394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.060593 (+ 199us) server_negotiation.cc:407] Connection header received
0810 04:20:08.061148 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.061149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.061162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.061218 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:08.062014 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.062527 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.063891 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.064154 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.064259 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.065280 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:08.065282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:08.065282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:08.065286 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:08.065300 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:08.065304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:08.065347 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.065401 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.065422 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 04:20:08.065778 4258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.060181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:44699 (local address 127.15.216.67:56285)
0810 04:20:08.060464 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:08.060498 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:08.061034 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:08.061302 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:08.061306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:08.061317 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:08.061914 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.061919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.062632 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.062633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.063732 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.063736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.064319 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.064320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.064470 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.065147 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:08.065147 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:08.065175 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:08.065197 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:08.065349 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:08.065353 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:08.065355 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:20:08.065404 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 04:20:08.065483 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
I20260810 04:20:08.066308 4256 heartbeater.cc:347] Connected to a master server at 127.15.216.126:44699
I20260810 04:20:08.066403 4256 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:08.066592 4256 heartbeater.cc:511] Master 127.15.216.126:44699 requested a full tablet report, sending...
I20260810 04:20:08.067003 3740 ts_manager.cc:194] Registered new tserver with Master: 034faa4b18bb4846a657553c523a983b (127.15.216.67:35563)
I20260810 04:20:08.067473 3740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.67:56285
I20260810 04:20:08.075804 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:20:08.080412 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.077233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.15.216.126:44699)
0810 04:20:08.077328 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.077329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.077415 (+ 86us) server_negotiation.cc:407] Connection header received
0810 04:20:08.077457 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.077458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.077467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.077520 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:08.077748 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.078213 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.079031 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.079284 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.079350 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.080040 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:08.080042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:08.080042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:08.080046 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:08.080059 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:08.080063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:08.080260 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.080323 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.080336 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 04:20:08.082422 3740 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:20:08.083940 3740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:20:08.091336 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.086578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41303 (local address 127.0.0.1:48548)
0810 04:20:08.086813 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:08.086823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:08.086874 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:08.087179 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:08.087182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:08.087187 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:08.087286 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.087288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.088651 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.088652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.090051 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.090055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.091028 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.091029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.091109 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.091123 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:20:08.091200 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 04:20:08.091253 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":11,"thread_start_us":68,"threads_started":1}
I20260810 04:20:08.091385 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.086692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.15.216.66:41303)
0810 04:20:08.086978 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.086984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.086996 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:20:08.087044 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.087047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.087064 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.087159 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:08.087378 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.088569 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.090169 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.090938 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.091001 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.091068 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.091253 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.091297 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260810 04:20:08.091931 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.086610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.15.216.65:33329)
0810 04:20:08.086997 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.087001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.087013 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:20:08.087052 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.087054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.087069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.087168 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:08.088699 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.089437 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.090807 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.091594 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.091681 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.091781 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.091813 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.091849 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":101,"threads_started":1}
I20260810 04:20:08.091924 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.086364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33329 (local address 127.0.0.1:55994)
0810 04:20:08.086718 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:08.086761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:08.086830 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:08.087179 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:08.087182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:08.087184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:08.087287 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.087288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.089551 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.089553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.090701 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.090705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.091695 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.091696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.091756 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.091760 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:08.091803 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 04:20:08.091841 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":50,"spinlock_wait_cycles":768,"thread_start_us":162,"threads_started":1}
I20260810 04:20:08.092331 4041 tablet_service.cc:1467] Processing CreateTablet for tablet 3a346746a9f14ac7975619c408f5ed9a (DEFAULT_TABLE table=test-table [id=9e416b74994340e7b0a8af25bfee758a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:08.092689 3889 tablet_service.cc:1467] Processing CreateTablet for tablet 3a346746a9f14ac7975619c408f5ed9a (DEFAULT_TABLE table=test-table [id=9e416b74994340e7b0a8af25bfee758a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:08.092811 4041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a346746a9f14ac7975619c408f5ed9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:08.092847 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.086362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:35563 (local address 127.0.0.1:56438)
0810 04:20:08.086719 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:08.086762 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:08.086832 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:08.087487 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:08.087488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:08.087490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:08.087519 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.087519 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.088727 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.088728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.090051 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.090055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.092585 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.092586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.092656 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.092659 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:08.092710 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 04:20:08.092753 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":88,"thread_start_us":61,"threads_started":1}
I20260810 04:20:08.092923 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.086453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.15.216.67:35563)
0810 04:20:08.087301 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.087304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.087313 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:08.087376 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.087377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.087393 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.087486 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:08.087570 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.088637 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.091106 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.092486 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.092607 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.092696 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.092731 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.092776 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":121,"threads_started":1}
I20260810 04:20:08.093111 3889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a346746a9f14ac7975619c408f5ed9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:08.093873 4183 tablet_service.cc:1467] Processing CreateTablet for tablet 3a346746a9f14ac7975619c408f5ed9a (DEFAULT_TABLE table=test-table [id=9e416b74994340e7b0a8af25bfee758a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:08.094252 4183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a346746a9f14ac7975619c408f5ed9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:08.096568 4270 tablet_bootstrap.cc:492] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b: Bootstrap starting.
I20260810 04:20:08.097311 4270 tablet_bootstrap.cc:654] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:08.097692 4270 log.cc:824] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:08.097818 4272 tablet_bootstrap.cc:492] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Bootstrap starting.
I20260810 04:20:08.098147 4271 tablet_bootstrap.cc:492] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Bootstrap starting.
I20260810 04:20:08.098567 4272 tablet_bootstrap.cc:654] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:08.098840 4271 tablet_bootstrap.cc:654] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:08.098923 4272 log.cc:824] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:08.099220 4271 log.cc:824] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:08.099814 4272 tablet_bootstrap.cc:492] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: No bootstrap required, opened a new log
I20260810 04:20:08.099879 4272 ts_tablet_manager.cc:1397] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 04:20:08.100041 4270 tablet_bootstrap.cc:492] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b: No bootstrap required, opened a new log
I20260810 04:20:08.100127 4270 ts_tablet_manager.cc:1397] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 04:20:08.102139 4272 raft_consensus.cc:348] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.102283 4272 raft_consensus.cc:374] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:08.102309 4272 raft_consensus.cc:729] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034faa4b18bb4846a657553c523a983b, State: Initialized, Role: FOLLOWER
I20260810 04:20:08.102448 4272 consensus_queue.cc:260] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.102387 4270 raft_consensus.cc:348] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.102594 4270 raft_consensus.cc:374] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:08.102638 4270 raft_consensus.cc:729] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b815f84640eb406aa6b8f05292845d7b, State: Initialized, Role: FOLLOWER
I20260810 04:20:08.102756 4272 ts_tablet_manager.cc:1428] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:20:08.102800 4256 heartbeater.cc:503] Master 127.15.216.126:44699 was elected leader, sending a full tablet report...
I20260810 04:20:08.102761 4270 consensus_queue.cc:260] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.103036 4270 ts_tablet_manager.cc:1428] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 04:20:08.103154 3968 heartbeater.cc:503] Master 127.15.216.126:44699 was elected leader, sending a full tablet report...
I20260810 04:20:08.103435 4271 tablet_bootstrap.cc:492] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: No bootstrap required, opened a new log
I20260810 04:20:08.103513 4271 ts_tablet_manager.cc:1397] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 04:20:08.105777 4271 raft_consensus.cc:348] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.105966 4271 raft_consensus.cc:374] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:08.106010 4271 raft_consensus.cc:729] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec64f39139748de80d2c6b0f68b6da6, State: Initialized, Role: FOLLOWER
I20260810 04:20:08.106129 4271 consensus_queue.cc:260] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.106423 4271 ts_tablet_manager.cc:1428] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 04:20:08.106500 4113 heartbeater.cc:503] Master 127.15.216.126:44699 was elected leader, sending a full tablet report...
W20260810 04:20:08.171731 4114 tablet.cc:2369] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:20:08.192094 4276 raft_consensus.cc:482] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:08.192224 4276 raft_consensus.cc:504] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.192581 4276 leader_election.cc:288] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b815f84640eb406aa6b8f05292845d7b (127.15.216.65:33329), 2ec64f39139748de80d2c6b0f68b6da6 (127.15.216.66:41303)
I20260810 04:20:08.195799 4258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.192762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:41303 (local address 127.15.216.67:60733)
0810 04:20:08.192836 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:08.192847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:08.192893 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:08.193064 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:08.193065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:08.193067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:08.193144 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.193147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.193889 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.193891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.194737 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.194740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.195493 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.195494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.195589 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.195593 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:08.195657 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:20:08.195704 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":21}
I20260810 04:20:08.195827 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.192881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:60733 (local address 127.15.216.66:41303)
0810 04:20:08.192926 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.192928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.192936 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:08.192966 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.192967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.192985 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.193058 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:08.193230 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.193802 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.194840 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.195410 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.195487 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.195496 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.195704 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.195745 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:20:08.195827 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.192828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:42847 (local address 127.15.216.65:33329)
0810 04:20:08.192883 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.192885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.193047 (+ 162us) server_negotiation.cc:407] Connection header received
0810 04:20:08.193116 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.193118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.193129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.193208 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:08.193359 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.193886 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.194914 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.195473 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.195548 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.195554 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.195703 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.195765 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 04:20:08.195811 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.192763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33329 (local address 127.15.216.67:42847)
0810 04:20:08.192949 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:08.192959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:08.193056 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:08.193210 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:08.193211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:08.193212 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:08.193292 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.193294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.193977 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.193979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.194773 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:08.194775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.195550 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.195551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:08.195621 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.195625 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:08.195678 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 04:20:08.195704 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":23,"thread_start_us":101,"threads_started":1}
I20260810 04:20:08.196312 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a346746a9f14ac7975619c408f5ed9a" candidate_uuid: "034faa4b18bb4846a657553c523a983b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b815f84640eb406aa6b8f05292845d7b" is_pre_election: true
I20260810 04:20:08.196384 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a346746a9f14ac7975619c408f5ed9a" candidate_uuid: "034faa4b18bb4846a657553c523a983b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec64f39139748de80d2c6b0f68b6da6" is_pre_election: true
I20260810 04:20:08.196448 3914 raft_consensus.cc:2454] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034faa4b18bb4846a657553c523a983b in term 0.
I20260810 04:20:08.196570 4064 raft_consensus.cc:2454] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034faa4b18bb4846a657553c523a983b in term 0.
I20260810 04:20:08.196739 4131 leader_election.cc:302] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034faa4b18bb4846a657553c523a983b, b815f84640eb406aa6b8f05292845d7b; no voters:
I20260810 04:20:08.196897 4276 raft_consensus.cc:2781] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:20:08.196995 4276 raft_consensus.cc:482] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:20:08.197031 4276 raft_consensus.cc:3037] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:08.197837 4276 raft_consensus.cc:504] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.197999 4276 leader_election.cc:288] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [CANDIDATE]: Term 1 election: Requested vote from peers b815f84640eb406aa6b8f05292845d7b (127.15.216.65:33329), 2ec64f39139748de80d2c6b0f68b6da6 (127.15.216.66:41303)
I20260810 04:20:08.198235 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a346746a9f14ac7975619c408f5ed9a" candidate_uuid: "034faa4b18bb4846a657553c523a983b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b815f84640eb406aa6b8f05292845d7b"
I20260810 04:20:08.198236 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a346746a9f14ac7975619c408f5ed9a" candidate_uuid: "034faa4b18bb4846a657553c523a983b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec64f39139748de80d2c6b0f68b6da6"
I20260810 04:20:08.198319 4064 raft_consensus.cc:3037] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:08.198319 3914 raft_consensus.cc:3037] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:08.199220 3914 raft_consensus.cc:2454] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034faa4b18bb4846a657553c523a983b in term 1.
I20260810 04:20:08.199220 4064 raft_consensus.cc:2454] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034faa4b18bb4846a657553c523a983b in term 1.
I20260810 04:20:08.199381 4131 leader_election.cc:302] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034faa4b18bb4846a657553c523a983b, b815f84640eb406aa6b8f05292845d7b; no voters:
I20260810 04:20:08.199487 4276 raft_consensus.cc:2781] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:08.199656 4276 raft_consensus.cc:686] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 1 LEADER]: Becoming Leader. State: Replica: 034faa4b18bb4846a657553c523a983b, State: Running, Role: LEADER
I20260810 04:20:08.199774 4276 consensus_queue.cc:237] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } }
I20260810 04:20:08.200716 3739 catalog_manager.cc:5495] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b reported cstate change: term changed from 0 to 1, leader changed from <none> to 034faa4b18bb4846a657553c523a983b (127.15.216.67). New cstate: current_term: 1 leader_uuid: "034faa4b18bb4846a657553c523a983b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034faa4b18bb4846a657553c523a983b" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 35563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 } health_report { overall_health: UNKNOWN } } }
I20260810 04:20:08.207708 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.205420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.15.216.67:35563)
0810 04:20:08.205482 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:08.205484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:08.205493 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:08.205530 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:08.205530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:08.205546 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:08.205669 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:08.205787 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.206242 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.207039 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:08.207223 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:08.207281 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:08.207419 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:08.207501 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:08.207570 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:08.207623 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 04:20:08.207632 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:20:08.209556 3914 raft_consensus.cc:1261] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 1 FOLLOWER]: Refusing update from remote peer 034faa4b18bb4846a657553c523a983b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:08.209569 4064 raft_consensus.cc:1261] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 1 FOLLOWER]: Refusing update from remote peer 034faa4b18bb4846a657553c523a983b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:08.209945 4281 consensus_queue.cc:1035] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec64f39139748de80d2c6b0f68b6da6" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:08.210103 4276 consensus_queue.cc:1035] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b815f84640eb406aa6b8f05292845d7b" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:08.212601 4285 mvcc.cc:205] Tried to move back new op lower bound from 7316830651223597056 to 7316830651187060736. Current Snapshot: MvccSnapshot[applied={T|T < 7316830651223597056}]
I20260810 04:20:08.213446 4286 mvcc.cc:205] Tried to move back new op lower bound from 7316830651223597056 to 7316830651187060736. Current Snapshot: MvccSnapshot[applied={T|T < 7316830651223597056}]
I20260810 04:20:08.215101 3739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:08.215190 3739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:08.216003 3739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9562cb40ce3343149a70042d0802f736: Sending DeleteTablet for 3 replicas of tablet 3a346746a9f14ac7975619c408f5ed9a
I20260810 04:20:08.216332 3889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a346746a9f14ac7975619c408f5ed9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:08 UTC) from {username='slave'} at 127.0.0.1:55994
I20260810 04:20:08.216437 4183 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a346746a9f14ac7975619c408f5ed9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:08 UTC) from {username='slave'} at 127.0.0.1:56438
I20260810 04:20:08.216661 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 3812
I20260810 04:20:08.216645 4041 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a346746a9f14ac7975619c408f5ed9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:08 UTC) from {username='slave'} at 127.0.0.1:48548
I20260810 04:20:08.216811 4294 tablet_replica.cc:330] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b: stopping tablet replica
I20260810 04:20:08.216962 4294 raft_consensus.cc:2229] T 3a346746a9f14ac7975619c408f5ed9a P b815f84640eb406aa6b8f05292845d7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:08.223783 4295 tablet_replica.cc:330] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: stopping tablet replica
I20260810 04:20:08.223912 4295 raft_consensus.cc:2229] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:20:08.224023 4295 raft_consensus.cc:2258] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:08.224196 4296 tablet_replica.cc:330] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: stopping tablet replica
I20260810 04:20:08.224318 4296 raft_consensus.cc:2229] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:08.224439 4296 raft_consensus.cc:2258] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:08.224702 4295 ts_tablet_manager.cc:1905] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:08.224931 4296 ts_tablet_manager.cc:1905] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:08.227013 4295 ts_tablet_manager.cc:1918] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:08.227027 4296 ts_tablet_manager.cc:1918] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:08.227099 4296 log.cc:1196] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/3a346746a9f14ac7975619c408f5ed9a
I20260810 04:20:08.227270 4295 log.cc:1196] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/3a346746a9f14ac7975619c408f5ed9a
I20260810 04:20:08.227376 4296 ts_tablet_manager.cc:1939] T 3a346746a9f14ac7975619c408f5ed9a P 2ec64f39139748de80d2c6b0f68b6da6: Deleting consensus metadata
I20260810 04:20:08.227681 4295 ts_tablet_manager.cc:1939] T 3a346746a9f14ac7975619c408f5ed9a P 034faa4b18bb4846a657553c523a983b: Deleting consensus metadata
I20260810 04:20:08.228041 3724 catalog_manager.cc:4841] TS 2ec64f39139748de80d2c6b0f68b6da6 (127.15.216.66:41303): tablet 3a346746a9f14ac7975619c408f5ed9a (table test-table [id=9e416b74994340e7b0a8af25bfee758a]) successfully deleted
I20260810 04:20:08.228227 3724 catalog_manager.cc:4841] TS 034faa4b18bb4846a657553c523a983b (127.15.216.67:35563): tablet 3a346746a9f14ac7975619c408f5ed9a (table test-table [id=9e416b74994340e7b0a8af25bfee758a]) successfully deleted
W20260810 04:20:08.229173 3721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:33329 (error 108)
I20260810 04:20:08.229511 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 3972
I20260810 04:20:08.229736 4267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:08.229576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33329 (local address 127.0.0.1:44460)
0810 04:20:08.229654 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:08.229696 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 04:20:08.234615 3721 catalog_manager.cc:4584] TS b815f84640eb406aa6b8f05292845d7b (127.15.216.65:33329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a346746a9f14ac7975619c408f5ed9a: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33329: connect: Connection refused (error 111)
I20260810 04:20:08.241014 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 4116
I20260810 04:20:08.247745 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 3704
2026-08-10T04:20:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:08 dist-test-slave-g350 krb5kdc[4300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:08 dist-test-slave-g350 krb5kdc[4300](info): set up 2 sockets
Aug 10 04:20:08 dist-test-slave-g350 krb5kdc[4300](info): commencing operation
krb5kdc: starting...
W20260810 04:20:09.377957 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:09 dist-test-slave-g350 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:09.504660 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41659
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:41659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:09.587366 4316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:09.587615 4316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:09.587652 4316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:09.588825 4316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:09.588899 4316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:09.588936 4316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:09.588967 4316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:09.589013 4316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:09.590590 4316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:41659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:09.590878 4316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:09.591109 4316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:09.596252 4323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:09.596375 4316 server_base.cc:1034] running on GCE node
W20260810 04:20:09.596211 4322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:09.596215 4325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:09.596753 4316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:09.597095 4316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:09.598260 4316 hybrid_clock.cc:648] HybridClock initialized: now 1786335609598238 us; error 33 us; skew 500 ppm
Aug 10 04:20:09 dist-test-slave-g350 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335609, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:09.600732 4316 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:09.601159 4316 webserver.cc:460] Webserver started at http://127.15.216.126:40969/ using document root <none> and password file <none>
I20260810 04:20:09.601408 4316 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:09.601481 4316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:09.601709 4316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:09.602761 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "796ad50e151447558590604a2e98c5be"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "35546f719c17650ecdda2ff0dc4dc511"
server_key_iv: "8393b5e2d4db96e7922cc4dcb9da2b90"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.603139 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "796ad50e151447558590604a2e98c5be"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "35546f719c17650ecdda2ff0dc4dc511"
server_key_iv: "8393b5e2d4db96e7922cc4dcb9da2b90"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.604461 4316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:09.605180 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:09.605412 4316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:09.605515 4316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "796ad50e151447558590604a2e98c5be"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "35546f719c17650ecdda2ff0dc4dc511"
server_key_iv: "8393b5e2d4db96e7922cc4dcb9da2b90"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.605629 4316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:09.622546 4316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:09.624405 4316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:09.624577 4316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:09.628895 4316 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:41659
I20260810 04:20:09.628928 4391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:41659 every 8 connection(s)
I20260810 04:20:09.629287 4316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:09.629842 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 4316
I20260810 04:20:09.629976 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:09.630183 16225 external_mini_cluster.cc:1428] Setting key 1f7e455bb63d4f24e7f005daf667ef3b
I20260810 04:20:09.630445 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:09.633225 4392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be: Bootstrap starting.
I20260810 04:20:09.633909 4392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:09.634246 4392 log.cc:824] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:09.634980 4392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be: No bootstrap required, opened a new log
Aug 10 04:20:09 dist-test-slave-g350 krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:09.637316 4392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad50e151447558590604a2e98c5be" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41659 } }
I20260810 04:20:09.637501 4392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:09.637530 4392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796ad50e151447558590604a2e98c5be, State: Initialized, Role: FOLLOWER
I20260810 04:20:09.637713 4392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad50e151447558590604a2e98c5be" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41659 } }
I20260810 04:20:09.637781 4392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:09.637841 4392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:09.637908 4392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:09.638532 4392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad50e151447558590604a2e98c5be" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41659 } }
I20260810 04:20:09.638644 4392 leader_election.cc:302] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796ad50e151447558590604a2e98c5be; no voters:
I20260810 04:20:09.638787 4392 leader_election.cc:288] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:09.638841 4397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:09.638998 4397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [term 1 LEADER]: Becoming Leader. State: Replica: 796ad50e151447558590604a2e98c5be, State: Running, Role: LEADER
I20260810 04:20:09.639078 4392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:09.639088 4397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad50e151447558590604a2e98c5be" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41659 } }
I20260810 04:20:09.639497 4397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796ad50e151447558590604a2e98c5be. Latest consensus state: current_term: 1 leader_uuid: "796ad50e151447558590604a2e98c5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad50e151447558590604a2e98c5be" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41659 } } }
I20260810 04:20:09.639590 4397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:09.639506 4399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796ad50e151447558590604a2e98c5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796ad50e151447558590604a2e98c5be" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 41659 } } }
I20260810 04:20:09.639771 4399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:09.639808 4404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:09.640358 4404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:09.641541 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:09.630656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.15.216.126:41659)
0810 04:20:09.630879 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:09.630887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:09.630924 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:20:09.631559 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:09.631568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:09.631614 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:09.631762 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:09.632416 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:09.633321 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:09.634177 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:09.634672 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:09.634794 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:09.637466 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:09.637469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:09.637470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:09.637480 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:09.639645 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:09.640097 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:09.640099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:09.640100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:09.640199 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:09.640443 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:09.640444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:09.640445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:09.640848 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:09.640990 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:09.641170 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 04:20:09.641333 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":105,"threads_started":1}
I20260810 04:20:09.642629 4404 catalog_manager.cc:1347] Generated new cluster ID: 3a19fdcce8284eafbc5e695469811100
I20260810 04:20:09.642670 4404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:09.655613 4404 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:09.656278 4404 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:09.663990 4404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be: Generated new TSK 0
I20260810 04:20:09.664199 4404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.216.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
Entry for principal kudu/127.15.216.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab.
I20260810 04:20:09.692001 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41659
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:09.777539 4418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:09.777791 4418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:09.777817 4418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:09.778977 4418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:09.779021 4418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:09.779048 4418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:20:09.780571 4418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.65.keytab
--principal=kudu/127.15.216.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:09.780831 4418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:09.781073 4418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:09.786448 4424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:09.786455 4426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:09.786469 4423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:09.786739 4418 server_base.cc:1034] running on GCE node
I20260810 04:20:09.786942 4418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:09.787137 4418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:09.788307 4418 hybrid_clock.cc:648] HybridClock initialized: now 1786335609788298 us; error 39 us; skew 500 ppm
Aug 10 04:20:09 dist-test-slave-g350 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335609, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:09.790907 4418 init.cc:377] Logged in from keytab as kudu/127.15.216.65@KRBTEST.COM (short username kudu)
I20260810 04:20:09.791342 4418 webserver.cc:460] Webserver started at http://127.15.216.65:38059/ using document root <none> and password file <none>
I20260810 04:20:09.791569 4418 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:09.791637 4418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:09.791790 4418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:09.792841 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "f121a27e0baf4b1988d34022f706b01f"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "f8733801a486f065113076401014e048"
server_key_iv: "cd0bb3159481fb28f3abfa5e5460aa8c"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.793213 4418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "f121a27e0baf4b1988d34022f706b01f"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "f8733801a486f065113076401014e048"
server_key_iv: "cd0bb3159481fb28f3abfa5e5460aa8c"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.794672 4418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:09.795464 4433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:09.795641 4418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:09.795751 4418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "f121a27e0baf4b1988d34022f706b01f"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "f8733801a486f065113076401014e048"
server_key_iv: "cd0bb3159481fb28f3abfa5e5460aa8c"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.795830 4418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:09.810869 4418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:09.812868 4418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:09.813042 4418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:09.813304 4418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:09.813704 4418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:09.813759 4418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:09.813800 4418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:09.813828 4418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:09.820048 4418 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:35017
I20260810 04:20:09.820128 4558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:35017 every 8 connection(s)
I20260810 04:20:09.820434 4418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
Aug 10 04:20:09 dist-test-slave-g350 krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335609, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.65@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:09.827984 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 4418
I20260810 04:20:09.828086 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:20:09.828374 16225 external_mini_cluster.cc:1428] Setting key d259122b8eacda4f3b1a5c6a3a3eca62
I20260810 04:20:09.831806 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:09.821184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:49075 (local address 127.15.216.126:41659)
0810 04:20:09.821259 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:09.821261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:09.821496 (+ 235us) server_negotiation.cc:407] Connection header received
0810 04:20:09.822132 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:09.822134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:09.822146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:09.822227 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:09.823244 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:09.823797 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:09.824780 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:09.824990 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:09.825048 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:09.827288 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:09.827289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:09.827289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:09.827296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:09.830278 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:09.830681 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:09.830683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:09.830683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:09.830751 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:09.831072 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:09.831073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:09.831074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:09.831228 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:09.831303 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:09.831660 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 04:20:09.831740 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 04:20:09.832139 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:09.821169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41659 (local address 127.15.216.65:49075)
0810 04:20:09.821356 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:09.821402 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:09.821973 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:09.822316 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:09.822319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:09.822920 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:09.823120 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:09.823125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:09.823890 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:09.823891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:09.824694 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:09.824696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:09.825070 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:09.825071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:09.825174 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:09.825775 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:09.825783 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:09.827202 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:09.830367 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:09.830374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:09.830377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:09.830603 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:09.830853 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:09.830854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:09.830855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:09.830986 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:09.831310 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:09.831320 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:09.831509 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 04:20:09.831638 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 04:20:09.831855 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
I20260810 04:20:09.832839 4559 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41659
I20260810 04:20:09.832948 4559 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:09.833149 4559 heartbeater.cc:511] Master 127.15.216.126:41659 requested a full tablet report, sending...
I20260810 04:20:09.833832 4351 ts_manager.cc:194] Registered new tserver with Master: f121a27e0baf4b1988d34022f706b01f (127.15.216.65:35017)
I20260810 04:20:09.834847 4351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.65@KRBTEST.COM'} at 127.15.216.65:49075
WARNING: no policy specified for kudu/127.15.216.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
Entry for principal kudu/127.15.216.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab.
I20260810 04:20:09.857095 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41659
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:09.948547 4564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:09.948753 4564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:09.948778 4564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:09.949934 4564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:09.949975 4564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:09.950001 4564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:20:09.951383 4564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.66.keytab
--principal=kudu/127.15.216.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:09.951567 4564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:09.951745 4564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:09.957434 4572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:09.957481 4564 server_base.cc:1034] running on GCE node
W20260810 04:20:09.957445 4570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:09.957479 4569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:09.957849 4564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:09.958112 4564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:09.959277 4564 hybrid_clock.cc:648] HybridClock initialized: now 1786335609959259 us; error 30 us; skew 500 ppm
Aug 10 04:20:09 dist-test-slave-g350 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335609, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:09.961827 4564 init.cc:377] Logged in from keytab as kudu/127.15.216.66@KRBTEST.COM (short username kudu)
I20260810 04:20:09.962217 4564 webserver.cc:460] Webserver started at http://127.15.216.66:35509/ using document root <none> and password file <none>
I20260810 04:20:09.962466 4564 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:09.962514 4564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:09.962685 4564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:09.963688 4564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "92b862fc066a4c7488fa9330a42ac5d1"
server_key_iv: "e4c6655423c989205790bcc1ac93ed95"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.963992 4564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "92b862fc066a4c7488fa9330a42ac5d1"
server_key_iv: "e4c6655423c989205790bcc1ac93ed95"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.965391 4564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:09.966264 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:09.966501 4564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:20:09.966572 4564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2"
format_stamp: "Formatted at 2026-08-10 04:20:09 on dist-test-slave-g350"
server_key: "92b862fc066a4c7488fa9330a42ac5d1"
server_key_iv: "e4c6655423c989205790bcc1ac93ed95"
server_key_version: "encryptionkey@0"
I20260810 04:20:09.966625 4564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:09.990088 4564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:09.992210 4564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:09.992336 4564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:09.992587 4564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:09.992955 4564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:09.993036 4564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:09.993080 4564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:09.993104 4564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:09.999959 4564 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:46251
I20260810 04:20:10.000021 4707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:46251 every 8 connection(s)
I20260810 04:20:10.000372 4564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:20:10.003317 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 4564
I20260810 04:20:10.003448 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:20:10.003618 16225 external_mini_cluster.cc:1428] Setting key b89248d62c40665ea2d0b91a8e00effb
Aug 10 04:20:10 dist-test-slave-g350 krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335609, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.66@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:10.011618 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.001158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:57217 (local address 127.15.216.126:41659)
0810 04:20:10.001248 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.001250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.001449 (+ 199us) server_negotiation.cc:407] Connection header received
0810 04:20:10.002022 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.002023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.002033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.002093 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:10.003126 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.003950 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.005119 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.005436 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.005518 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.008351 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:10.008357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:10.008358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:10.008366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:10.010128 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:10.010589 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:10.010590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:10.010591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:10.010654 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:10.010922 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:10.010923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:10.010923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:10.011043 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:10.011123 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.011426 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.011522 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 04:20:10.011832 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.001118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41659 (local address 127.15.216.66:57217)
0810 04:20:10.001310 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.001355 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.001933 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.002146 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.002149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.002828 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:10.003027 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.003032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.004064 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.004064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.005027 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.005030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.005554 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.005555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.005765 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.006374 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:10.006386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:10.008233 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:10.010222 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:10.010231 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:10.010233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:10.010505 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:10.010731 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:10.010732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:10.010732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:10.010845 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:10.011114 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:10.011121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:10.011272 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.011403 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.011574 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260810 04:20:10.012531 4709 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41659
I20260810 04:20:10.012646 4709 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:10.012835 4709 heartbeater.cc:511] Master 127.15.216.126:41659 requested a full tablet report, sending...
I20260810 04:20:10.013324 4351 ts_manager.cc:194] Registered new tserver with Master: 1e8052e7e6894e5aa63f2e032f0d9bc2 (127.15.216.66:46251)
WARNING: no policy specified for kudu/127.15.216.67@KRBTEST.COM; defaulting to no policy
I20260810 04:20:10.013917 4351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.66@KRBTEST.COM'} at 127.15.216.66:57217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
Entry for principal kudu/127.15.216.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab.
I20260810 04:20:10.032974 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41659
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:10.117498 4715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:10.117767 4715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:10.117794 4715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:10.118860 4715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:10.118894 4715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:10.118919 4715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:20:10.120325 4715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.67.keytab
--principal=kudu/127.15.216.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:41659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:10.120508 4715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:10.120677 4715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:10.125783 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.125797 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.125882 4724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:10.125993 4715 server_base.cc:1034] running on GCE node
I20260810 04:20:10.126180 4715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:10.126384 4715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:10.127529 4715 hybrid_clock.cc:648] HybridClock initialized: now 1786335610127497 us; error 48 us; skew 500 ppm
Aug 10 04:20:10 dist-test-slave-g350 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335610, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:10.130112 4715 init.cc:377] Logged in from keytab as kudu/127.15.216.67@KRBTEST.COM (short username kudu)
I20260810 04:20:10.130568 4715 webserver.cc:460] Webserver started at http://127.15.216.67:36691/ using document root <none> and password file <none>
I20260810 04:20:10.130813 4715 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:10.130882 4715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:10.131031 4715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:10.132058 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "b186cef35a064ab09d40ec6306086185"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "6644bdc5e4b639b84c4ebb5bacb6411a"
server_key_iv: "fbb37b28ed83d94e0af4d28f7253a455"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.132411 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "b186cef35a064ab09d40ec6306086185"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "6644bdc5e4b639b84c4ebb5bacb6411a"
server_key_iv: "fbb37b28ed83d94e0af4d28f7253a455"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.133832 4715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:20:10.134629 4732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.134867 4715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:20:10.134986 4715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "b186cef35a064ab09d40ec6306086185"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "6644bdc5e4b639b84c4ebb5bacb6411a"
server_key_iv: "fbb37b28ed83d94e0af4d28f7253a455"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.135069 4715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:10.153807 4715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:10.155817 4715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:10.156010 4715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:10.156270 4715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:10.156584 4715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:10.156641 4715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.156682 4715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:10.156711 4715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.163197 4715 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:42161
I20260810 04:20:10.163254 4859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:42161 every 8 connection(s)
I20260810 04:20:10.163569 4715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:20:10.169096 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 4715
I20260810 04:20:10.169214 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:20:10.169425 16225 external_mini_cluster.cc:1428] Setting key 4c6e97efce9c139266649171869c6b30
Aug 10 04:20:10 dist-test-slave-g350 krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335610, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.67@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:10.173936 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.164452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:57903 (local address 127.15.216.126:41659)
0810 04:20:10.164511 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.164513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.164713 (+ 200us) server_negotiation.cc:407] Connection header received
0810 04:20:10.165183 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.165184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.165193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.165258 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:10.166357 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.166856 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.167842 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.168029 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.168087 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.170930 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:10.170932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:10.170932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:10.170940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:10.172652 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:10.173041 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:10.173042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:10.173042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:10.173093 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:10.173331 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:10.173332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:10.173333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:10.173451 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:10.173516 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.173806 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.173872 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:20:10.174194 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.164322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:41659 (local address 127.15.216.67:57903)
0810 04:20:10.164606 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.164639 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.165105 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.165312 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.165315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.166004 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:10.166223 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.166228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.166942 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.166943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.167739 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.167742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.168105 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.168106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.168227 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.168793 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:10.168804 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:10.170821 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:10.172726 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:10.172733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:10.172735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:10.172973 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:10.173152 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:10.173152 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:10.173152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:10.173250 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:10.173514 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:10.173519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:10.173695 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.173785 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.173925 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260810 04:20:10.174705 4860 heartbeater.cc:347] Connected to a master server at 127.15.216.126:41659
I20260810 04:20:10.174806 4860 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:10.174985 4860 heartbeater.cc:511] Master 127.15.216.126:41659 requested a full tablet report, sending...
I20260810 04:20:10.175361 4351 ts_manager.cc:194] Registered new tserver with Master: b186cef35a064ab09d40ec6306086185 (127.15.216.67:42161)
I20260810 04:20:10.175696 4351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.216.67@KRBTEST.COM'} at 127.15.216.67:57903
I20260810 04:20:10.181691 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:20:10.189801 4395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.183215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.15.216.126:41659)
0810 04:20:10.183391 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.183393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.183426 (+ 33us) server_negotiation.cc:407] Connection header received
0810 04:20:10.183485 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.183485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.183494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.183559 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:10.184108 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.184638 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.185317 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.185492 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.185544 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.186791 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:10.186793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:10.186793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:10.186800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:10.188287 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:10.188639 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:10.188640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:10.188640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:10.188692 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:10.188936 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:10.188937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:10.188938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:10.189175 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:10.189369 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.189563 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.189683 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260810 04:20:10.191325 4351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:20:10.192929 4351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:20:10.200022 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.195636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.15.216.66:46251)
0810 04:20:10.195851 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.195857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.196076 (+ 219us) server_negotiation.cc:407] Connection header received
0810 04:20:10.196125 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.196128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.196145 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.196221 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:10.196422 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.197564 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.198991 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.199585 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.199675 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.199735 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.199898 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.199938 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260810 04:20:10.200011 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.195475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:46251 (local address 127.0.0.1:43986)
0810 04:20:10.195826 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.195890 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.195977 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.196266 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.196269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.196271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:10.196339 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.196340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.197696 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.197697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.198885 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.198889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.199678 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.199679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.199767 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.199781 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.199878 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.199935 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":23,"thread_start_us":75,"threads_started":1}
I20260810 04:20:10.200161 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.195636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.15.216.65:35017)
0810 04:20:10.195877 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.195881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.196094 (+ 213us) server_negotiation.cc:407] Connection header received
0810 04:20:10.196139 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.196141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.196157 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.196229 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:10.196448 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.197358 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.198991 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.199713 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.199772 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.199820 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.200047 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.200086 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":124,"threads_started":1}
I20260810 04:20:10.200286 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.195687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:42161 (local address 127.0.0.1:37934)
0810 04:20:10.196486 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.196497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.196547 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.196730 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.196731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.196733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:10.196777 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.196778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.197986 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.197987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.198838 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.198841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.199816 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.199817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.199912 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.199916 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.199963 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.200001 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260810 04:20:10.200542 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.195475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:35017 (local address 127.0.0.1:39768)
0810 04:20:10.195951 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.195963 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.196029 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.196314 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.196315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.196317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:10.196373 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.196374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.197463 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.197464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.198926 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.198928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.199848 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.199849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.199900 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.199903 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.199957 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.200000 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":33,"thread_start_us":79,"threads_started":1}
I20260810 04:20:10.200196 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.195795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.15.216.67:42161)
0810 04:20:10.195988 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.195992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.196564 (+ 572us) server_negotiation.cc:407] Connection header received
0810 04:20:10.196613 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.196616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.196633 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.196713 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:10.196834 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.197869 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.198933 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.199648 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.199720 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.199778 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.200071 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.200115 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260810 04:20:10.201200 4487 tablet_service.cc:1467] Processing CreateTablet for tablet b54daf0c1b5b4443b0621e8b0c2f3dce (DEFAULT_TABLE table=test-table [id=116d7d86e4d54f6fbe95ad7c6e3b8d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:10.201257 4787 tablet_service.cc:1467] Processing CreateTablet for tablet b54daf0c1b5b4443b0621e8b0c2f3dce (DEFAULT_TABLE table=test-table [id=116d7d86e4d54f6fbe95ad7c6e3b8d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:10.201239 4634 tablet_service.cc:1467] Processing CreateTablet for tablet b54daf0c1b5b4443b0621e8b0c2f3dce (DEFAULT_TABLE table=test-table [id=116d7d86e4d54f6fbe95ad7c6e3b8d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:10.201565 4487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54daf0c1b5b4443b0621e8b0c2f3dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:10.201584 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54daf0c1b5b4443b0621e8b0c2f3dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:10.201584 4634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54daf0c1b5b4443b0621e8b0c2f3dce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:10.204208 4878 tablet_bootstrap.cc:492] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f: Bootstrap starting.
I20260810 04:20:10.204856 4879 tablet_bootstrap.cc:492] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Bootstrap starting.
I20260810 04:20:10.205026 4878 tablet_bootstrap.cc:654] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:10.205072 4880 tablet_bootstrap.cc:492] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Bootstrap starting.
I20260810 04:20:10.205564 4879 tablet_bootstrap.cc:654] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:10.205636 4878 log.cc:824] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:10.205706 4880 tablet_bootstrap.cc:654] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:10.206174 4879 log.cc:824] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:10.206346 4880 log.cc:824] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:10.206508 4878 tablet_bootstrap.cc:492] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f: No bootstrap required, opened a new log
I20260810 04:20:10.206607 4878 ts_tablet_manager.cc:1397] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:20:10.206995 4879 tablet_bootstrap.cc:492] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: No bootstrap required, opened a new log
I20260810 04:20:10.207063 4879 ts_tablet_manager.cc:1397] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 04:20:10.207123 4880 tablet_bootstrap.cc:492] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: No bootstrap required, opened a new log
I20260810 04:20:10.207187 4880 ts_tablet_manager.cc:1397] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:20:10.208664 4878 raft_consensus.cc:348] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.208873 4878 raft_consensus.cc:374] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:10.208920 4878 raft_consensus.cc:729] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f121a27e0baf4b1988d34022f706b01f, State: Initialized, Role: FOLLOWER
I20260810 04:20:10.208886 4880 raft_consensus.cc:348] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.209076 4880 raft_consensus.cc:374] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:10.209051 4878 consensus_queue.cc:260] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.209126 4880 raft_consensus.cc:729] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b186cef35a064ab09d40ec6306086185, State: Initialized, Role: FOLLOWER
I20260810 04:20:10.209085 4879 raft_consensus.cc:348] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.209234 4880 consensus_queue.cc:260] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.209267 4879 raft_consensus.cc:374] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:10.209316 4879 raft_consensus.cc:729] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8052e7e6894e5aa63f2e032f0d9bc2, State: Initialized, Role: FOLLOWER
I20260810 04:20:10.209336 4878 ts_tablet_manager.cc:1428] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 04:20:10.209447 4559 heartbeater.cc:503] Master 127.15.216.126:41659 was elected leader, sending a full tablet report...
I20260810 04:20:10.209434 4879 consensus_queue.cc:260] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.209537 4880 ts_tablet_manager.cc:1428] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 04:20:10.209671 4860 heartbeater.cc:503] Master 127.15.216.126:41659 was elected leader, sending a full tablet report...
I20260810 04:20:10.209722 4879 ts_tablet_manager.cc:1428] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 04:20:10.209846 4709 heartbeater.cc:503] Master 127.15.216.126:41659 was elected leader, sending a full tablet report...
W20260810 04:20:10.250985 4710 tablet.cc:2369] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:10.321271 4560 tablet.cc:2369] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:20:10.338359 4885 raft_consensus.cc:482] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:10.338493 4885 raft_consensus.cc:504] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.339001 4885 leader_election.cc:288] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f121a27e0baf4b1988d34022f706b01f (127.15.216.65:35017), 1e8052e7e6894e5aa63f2e032f0d9bc2 (127.15.216.66:46251)
I20260810 04:20:10.343065 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.339197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:46251 (local address 127.15.216.67:55973)
0810 04:20:10.339289 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.339302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.339357 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.339556 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.339557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.339561 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:10.339657 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.339659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.340628 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.340629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.341964 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.341968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.342770 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.342771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.342855 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.342857 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.342917 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.342965 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":58}
I20260810 04:20:10.343086 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.339226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:35017 (local address 127.15.216.67:54523)
0810 04:20:10.339401 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.339412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.339459 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.339600 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.339600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.339601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:10.339666 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.339667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.340620 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.340622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.341981 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.341983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.342770 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.342771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.342849 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.342853 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.342928 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.342965 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":72,"thread_start_us":106,"threads_started":1}
I20260810 04:20:10.343112 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.339416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:54523 (local address 127.15.216.65:35017)
0810 04:20:10.339462 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.339464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.339499 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:20:10.339525 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.339526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.339538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.339587 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:10.339775 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.340518 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.342070 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.342678 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.342765 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.342774 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.342989 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.343041 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:20:10.343120 4873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.339332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:55973 (local address 127.15.216.66:46251)
0810 04:20:10.339414 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.339416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.339427 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:20:10.339459 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.339460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.339475 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.339540 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:10.339757 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.340518 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.342070 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.342649 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.342765 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.342774 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.342985 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.343036 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:20:10.343475 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54daf0c1b5b4443b0621e8b0c2f3dce" candidate_uuid: "b186cef35a064ab09d40ec6306086185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f121a27e0baf4b1988d34022f706b01f" is_pre_election: true
I20260810 04:20:10.343480 4656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54daf0c1b5b4443b0621e8b0c2f3dce" candidate_uuid: "b186cef35a064ab09d40ec6306086185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" is_pre_election: true
I20260810 04:20:10.343641 4507 raft_consensus.cc:2454] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b186cef35a064ab09d40ec6306086185 in term 0.
I20260810 04:20:10.343660 4656 raft_consensus.cc:2454] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b186cef35a064ab09d40ec6306086185 in term 0.
I20260810 04:20:10.343892 4735 leader_election.cc:302] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b186cef35a064ab09d40ec6306086185, f121a27e0baf4b1988d34022f706b01f; no voters:
I20260810 04:20:10.344038 4885 raft_consensus.cc:2781] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:20:10.344115 4885 raft_consensus.cc:482] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:20:10.344146 4885 raft_consensus.cc:3037] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:10.344883 4885 raft_consensus.cc:504] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.345032 4885 leader_election.cc:288] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [CANDIDATE]: Term 1 election: Requested vote from peers f121a27e0baf4b1988d34022f706b01f (127.15.216.65:35017), 1e8052e7e6894e5aa63f2e032f0d9bc2 (127.15.216.66:46251)
I20260810 04:20:10.345184 4656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54daf0c1b5b4443b0621e8b0c2f3dce" candidate_uuid: "b186cef35a064ab09d40ec6306086185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2"
I20260810 04:20:10.345192 4507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54daf0c1b5b4443b0621e8b0c2f3dce" candidate_uuid: "b186cef35a064ab09d40ec6306086185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f121a27e0baf4b1988d34022f706b01f"
I20260810 04:20:10.345284 4656 raft_consensus.cc:3037] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:10.345281 4507 raft_consensus.cc:3037] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:10.346096 4656 raft_consensus.cc:2454] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b186cef35a064ab09d40ec6306086185 in term 1.
I20260810 04:20:10.346174 4507 raft_consensus.cc:2454] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b186cef35a064ab09d40ec6306086185 in term 1.
I20260810 04:20:10.346369 4734 leader_election.cc:302] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8052e7e6894e5aa63f2e032f0d9bc2, b186cef35a064ab09d40ec6306086185; no voters:
I20260810 04:20:10.346539 4885 raft_consensus.cc:2781] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:10.346748 4885 raft_consensus.cc:686] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 1 LEADER]: Becoming Leader. State: Replica: b186cef35a064ab09d40ec6306086185, State: Running, Role: LEADER
I20260810 04:20:10.346868 4885 consensus_queue.cc:237] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } }
I20260810 04:20:10.347777 4351 catalog_manager.cc:5495] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 reported cstate change: term changed from 0 to 1, leader changed from <none> to b186cef35a064ab09d40ec6306086185 (127.15.216.67). New cstate: current_term: 1 leader_uuid: "b186cef35a064ab09d40ec6306086185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b186cef35a064ab09d40ec6306086185" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 42161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 } health_report { overall_health: UNKNOWN } } }
I20260810 04:20:10.381994 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.379204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.15.216.67:42161)
0810 04:20:10.379272 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.379275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.379284 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:10.379331 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.379332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.379354 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.379441 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:10.379600 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.380081 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.380862 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.381085 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.381163 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.381527 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:10.381678 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:10.381837 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.381857 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.381881 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:20:10.384143 4656 raft_consensus.cc:1261] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 1 FOLLOWER]: Refusing update from remote peer b186cef35a064ab09d40ec6306086185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:10.384160 4507 raft_consensus.cc:1261] T b54daf0c1b5b4443b0621e8b0c2f3dce P f121a27e0baf4b1988d34022f706b01f [term 1 FOLLOWER]: Refusing update from remote peer b186cef35a064ab09d40ec6306086185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:10.384594 4888 consensus_queue.cc:1035] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f121a27e0baf4b1988d34022f706b01f" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 35017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:10.384757 4885 consensus_queue.cc:1035] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8052e7e6894e5aa63f2e032f0d9bc2" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:10.387323 4892 mvcc.cc:205] Tried to move back new op lower bound from 7316830660130328576 to 7316830659981381632. Current Snapshot: MvccSnapshot[applied={T|T < 7316830660130328576}]
I20260810 04:20:10.389947 4351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:10.390041 4351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:10.390826 4351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 796ad50e151447558590604a2e98c5be: Sending DeleteTablet for 3 replicas of tablet b54daf0c1b5b4443b0621e8b0c2f3dce
I20260810 04:20:10.391274 4634 tablet_service.cc:1514] Processing DeleteTablet for tablet b54daf0c1b5b4443b0621e8b0c2f3dce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:10 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:43986
I20260810 04:20:10.391317 4787 tablet_service.cc:1514] Processing DeleteTablet for tablet b54daf0c1b5b4443b0621e8b0c2f3dce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:10 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:37934
I20260810 04:20:10.391400 4487 tablet_service.cc:1514] Processing DeleteTablet for tablet b54daf0c1b5b4443b0621e8b0c2f3dce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:10 UTC) from {username='kudu', principal='kudu/127.15.216.126@KRBTEST.COM'} at 127.0.0.1:39768
I20260810 04:20:10.391535 4902 tablet_replica.cc:330] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: stopping tablet replica
I20260810 04:20:10.391633 4902 raft_consensus.cc:2229] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:20:10.391726 4902 raft_consensus.cc:2258] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:10.391791 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 4418
I20260810 04:20:10.391836 4903 tablet_replica.cc:330] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: stopping tablet replica
I20260810 04:20:10.392045 4903 raft_consensus.cc:2229] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:10.392227 4903 raft_consensus.cc:2258] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:10.392289 4902 ts_tablet_manager.cc:1905] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:10.393314 4903 ts_tablet_manager.cc:1905] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:10.394515 4902 ts_tablet_manager.cc:1918] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:10.394610 4902 log.cc:1196] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/b54daf0c1b5b4443b0621e8b0c2f3dce
I20260810 04:20:10.394950 4902 ts_tablet_manager.cc:1939] T b54daf0c1b5b4443b0621e8b0c2f3dce P b186cef35a064ab09d40ec6306086185: Deleting consensus metadata
I20260810 04:20:10.395401 4337 catalog_manager.cc:4841] TS b186cef35a064ab09d40ec6306086185 (127.15.216.67:42161): tablet b54daf0c1b5b4443b0621e8b0c2f3dce (table test-table [id=116d7d86e4d54f6fbe95ad7c6e3b8d70]) successfully deleted
I20260810 04:20:10.395560 4903 ts_tablet_manager.cc:1918] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:10.395666 4903 log.cc:1196] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/b54daf0c1b5b4443b0621e8b0c2f3dce
I20260810 04:20:10.395979 4903 ts_tablet_manager.cc:1939] T b54daf0c1b5b4443b0621e8b0c2f3dce P 1e8052e7e6894e5aa63f2e032f0d9bc2: Deleting consensus metadata
I20260810 04:20:10.396417 4336 catalog_manager.cc:4841] TS 1e8052e7e6894e5aa63f2e032f0d9bc2 (127.15.216.66:46251): tablet b54daf0c1b5b4443b0621e8b0c2f3dce (table test-table [id=116d7d86e4d54f6fbe95ad7c6e3b8d70]) successfully deleted
W20260810 04:20:10.402091 4337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:35017 (error 108)
I20260810 04:20:10.402343 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 4564
I20260810 04:20:10.404003 4875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.403837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:35017 (local address 127.0.0.1:39780)
0810 04:20:10.403908 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.403956 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 04:20:10.404984 4337 catalog_manager.cc:4584] TS f121a27e0baf4b1988d34022f706b01f (127.15.216.65:35017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b54daf0c1b5b4443b0621e8b0c2f3dce: Network error: Client connection negotiation failed: client connection to 127.15.216.65:35017: connect: Connection refused (error 111)
I20260810 04:20:10.410209 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 4715
I20260810 04:20:10.418463 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 4316
2026-08-10T04:20:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2172 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T04:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:10Z Disabled control of system clock
I20260810 04:20:10.450208 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:37621
--webserver_interface=127.15.216.126
--webserver_port=0
--builtin_ntp_servers=127.15.216.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:37621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:20:10.531405 4910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:10.531616 4910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:10.531636 4910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:10.532673 4910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:10.532709 4910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:10.532722 4910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:10.532735 4910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:10.532747 4910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:10.534192 4910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:37621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:37621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:10.534420 4910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:10.534699 4910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:20:10.539673 4910 server_base.cc:1034] running on GCE node
W20260810 04:20:10.539641 4915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.539659 4916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.539654 4918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:10.540182 4910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:10.540483 4910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:10.541662 4910 hybrid_clock.cc:648] HybridClock initialized: now 1786335610541648 us; error 33 us; skew 500 ppm
I20260810 04:20:10.542948 4910 webserver.cc:460] Webserver started at http://127.15.216.126:44007/ using document root <none> and password file <none>
I20260810 04:20:10.543228 4910 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:10.543301 4910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:10.543474 4910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:10.544553 4910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "af496bed22a441599d9606041fc47596"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "a30ac4ca7806ee6a37e51b399f75bab8"
server_key_iv: "1532dba8a8a0275f1fcfd5da54ea3675"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.544927 4910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "af496bed22a441599d9606041fc47596"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "a30ac4ca7806ee6a37e51b399f75bab8"
server_key_iv: "1532dba8a8a0275f1fcfd5da54ea3675"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.546406 4910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:10.547189 4925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.547407 4910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:10.547519 4910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "af496bed22a441599d9606041fc47596"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "a30ac4ca7806ee6a37e51b399f75bab8"
server_key_iv: "1532dba8a8a0275f1fcfd5da54ea3675"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.547596 4910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:10.567458 4910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:10.567804 4910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:10.568133 4910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:10.573066 4910 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:37621
I20260810 04:20:10.573096 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:37621 every 8 connection(s)
I20260810 04:20:10.573496 4910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:10.574550 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:10.575206 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 4910
I20260810 04:20:10.575322 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:10.575532 16225 external_mini_cluster.cc:1428] Setting key 8920eee0522cc4401dcf3113b55f9092
I20260810 04:20:10.577428 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596: Bootstrap starting.
I20260810 04:20:10.578088 4990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:10.578392 4990 log.cc:824] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:10.579123 4990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596: No bootstrap required, opened a new log
I20260810 04:20:10.581161 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.576101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.15.216.126:37621)
0810 04:20:10.576333 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.576340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.576366 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:20:10.576781 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.576787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.576845 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.577009 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:10.577263 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.578320 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.579134 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.579611 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.579743 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.580543 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:10.580547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:10.580547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:10.580720 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:10.580755 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:10.580766 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:10.580846 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.580913 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.580994 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
I20260810 04:20:10.581341 4990 raft_consensus.cc:348] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af496bed22a441599d9606041fc47596" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37621 } }
I20260810 04:20:10.581534 4990 raft_consensus.cc:374] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:10.581607 4990 raft_consensus.cc:729] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af496bed22a441599d9606041fc47596, State: Initialized, Role: FOLLOWER
I20260810 04:20:10.581763 4990 consensus_queue.cc:260] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af496bed22a441599d9606041fc47596" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37621 } }
I20260810 04:20:10.581875 4990 raft_consensus.cc:388] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:10.581910 4990 raft_consensus.cc:482] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:10.581961 4990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:10.582530 4990 raft_consensus.cc:504] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af496bed22a441599d9606041fc47596" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37621 } }
I20260810 04:20:10.582623 4990 leader_election.cc:302] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af496bed22a441599d9606041fc47596; no voters:
I20260810 04:20:10.582754 4990 leader_election.cc:288] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:10.582850 4995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:10.583016 4995 raft_consensus.cc:686] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [term 1 LEADER]: Becoming Leader. State: Replica: af496bed22a441599d9606041fc47596, State: Running, Role: LEADER
I20260810 04:20:10.583052 4990 sys_catalog.cc:564] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:10.583103 4995 consensus_queue.cc:237] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af496bed22a441599d9606041fc47596" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37621 } }
I20260810 04:20:10.583484 4997 sys_catalog.cc:455] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af496bed22a441599d9606041fc47596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af496bed22a441599d9606041fc47596" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37621 } } }
I20260810 04:20:10.583484 4998 sys_catalog.cc:455] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af496bed22a441599d9606041fc47596. Latest consensus state: current_term: 1 leader_uuid: "af496bed22a441599d9606041fc47596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af496bed22a441599d9606041fc47596" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 37621 } } }
I20260810 04:20:10.583600 4997 sys_catalog.cc:458] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:10.583606 4998 sys_catalog.cc:458] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:10.583824 5004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:10.584398 5004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:10.586001 5004 catalog_manager.cc:1347] Generated new cluster ID: f5ba234453c14acaa2ce5645de23e012
I20260810 04:20:10.586059 5004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:10.611079 5004 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:10.611723 5004 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:10.618052 5004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596: Generated new TSK 0
I20260810 04:20:10.618326 5004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:10.626492 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.65:0
--local_ip_for_outbound_sockets=127.15.216.65
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37621
--builtin_ntp_servers=127.15.216.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:20:10.714334 5016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:10.714533 5016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:10.714556 5016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:10.715701 5016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:10.715744 5016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:10.715770 5016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.65
I20260810 04:20:10.717142 5016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.216.65
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.65
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:10.717406 5016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:10.717762 5016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:10.723143 5023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.723157 5022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:10.723174 5016 server_base.cc:1034] running on GCE node
W20260810 04:20:10.723255 5025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:10.723613 5016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:10.723860 5016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:10.725013 5016 hybrid_clock.cc:648] HybridClock initialized: now 1786335610725002 us; error 28 us; skew 500 ppm
I20260810 04:20:10.726347 5016 webserver.cc:460] Webserver started at http://127.15.216.65:36303/ using document root <none> and password file <none>
I20260810 04:20:10.726586 5016 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:10.726629 5016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:10.726720 5016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:10.727761 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data/instance:
uuid: "90b3dee39218468eacb827c87fb72cbc"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "3410a7b18576324e54622bd147bc0152"
server_key_iv: "f55fb08bd9f8b6977c5aab202c4b0376"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.728061 5016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance:
uuid: "90b3dee39218468eacb827c87fb72cbc"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "3410a7b18576324e54622bd147bc0152"
server_key_iv: "f55fb08bd9f8b6977c5aab202c4b0376"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.729297 5016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:10.730077 5031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.730327 5016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:20:10.730425 5016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/wal
uuid: "90b3dee39218468eacb827c87fb72cbc"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "3410a7b18576324e54622bd147bc0152"
server_key_iv: "f55fb08bd9f8b6977c5aab202c4b0376"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.730481 5016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:10.742231 5016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:10.742501 5016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:10.742599 5016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:10.742790 5016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:10.743129 5016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:10.743163 5016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.743237 5016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:10.743263 5016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.749125 5016 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.65:33575
I20260810 04:20:10.749238 5158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.65:33575 every 8 connection(s)
I20260810 04:20:10.749572 5016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/data/info.pb
I20260810 04:20:10.753358 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 5016
I20260810 04:20:10.753471 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-0/wal/instance
I20260810 04:20:10.753738 16225 external_mini_cluster.cc:1428] Setting key 1e3a8d9baf5c18647e4801fb6d962b78
I20260810 04:20:10.754709 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.66:0
--local_ip_for_outbound_sockets=127.15.216.66
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37621
--builtin_ntp_servers=127.15.216.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 04:20:10.755494 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.750420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.65:59343 (local address 127.15.216.126:37621)
0810 04:20:10.750480 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.750482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.750732 (+ 250us) server_negotiation.cc:407] Connection header received
0810 04:20:10.751290 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.751292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.751303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.751355 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:10.752127 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.752667 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.753883 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.754114 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.754216 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.755157 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:10.755159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:10.755159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:10.755164 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:10.755180 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:10.755184 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:10.755238 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.755381 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.755412 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:20:10.755717 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.750407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:37621 (local address 127.15.216.65:59343)
0810 04:20:10.750585 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.750619 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.751120 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.751490 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.751493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.751501 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:10.752045 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.752049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.752756 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.752757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.753749 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.753752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.754247 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.754248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.754388 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.754988 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:10.754989 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:10.755029 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:10.755053 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:10.755284 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:10.755291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:10.755294 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.755382 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.755486 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260810 04:20:10.756814 5159 heartbeater.cc:347] Connected to a master server at 127.15.216.126:37621
I20260810 04:20:10.756925 5159 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:10.757112 5159 heartbeater.cc:511] Master 127.15.216.126:37621 requested a full tablet report, sending...
I20260810 04:20:10.757756 4947 ts_manager.cc:194] Registered new tserver with Master: 90b3dee39218468eacb827c87fb72cbc (127.15.216.65:33575)
I20260810 04:20:10.758384 4947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.65:59343
W20260810 04:20:10.840413 5162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:10.840646 5162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:10.840685 5162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:10.841953 5162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:10.842017 5162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:10.842058 5162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.66
I20260810 04:20:10.843415 5162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.216.66
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.66
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:10.843680 5162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:10.843987 5162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:10.849179 5167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.849166 5169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.849152 5171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:10.849416 5162 server_base.cc:1034] running on GCE node
I20260810 04:20:10.849702 5162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:10.849970 5162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:10.851115 5162 hybrid_clock.cc:648] HybridClock initialized: now 1786335610851094 us; error 29 us; skew 500 ppm
I20260810 04:20:10.852826 5162 webserver.cc:460] Webserver started at http://127.15.216.66:46467/ using document root <none> and password file <none>
I20260810 04:20:10.853255 5162 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:10.853317 5162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:10.853636 5162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:10.854816 5162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data/instance:
uuid: "004a357b0fac44ad8b2c4affde93ea31"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "e5584f6899824db4be906322e4b3d2c0"
server_key_iv: "e4eb0caf3ba3e3e6c0f50df436c2024e"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.855182 5162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance:
uuid: "004a357b0fac44ad8b2c4affde93ea31"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "e5584f6899824db4be906322e4b3d2c0"
server_key_iv: "e4eb0caf3ba3e3e6c0f50df436c2024e"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.856554 5162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:20:10.857306 5177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.857525 5162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:10.857652 5162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal
uuid: "004a357b0fac44ad8b2c4affde93ea31"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "e5584f6899824db4be906322e4b3d2c0"
server_key_iv: "e4eb0caf3ba3e3e6c0f50df436c2024e"
server_key_version: "encryptionkey@0"
I20260810 04:20:10.857748 5162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:10.874076 5162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:10.874449 5162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:10.874622 5162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:10.874889 5162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:10.875270 5162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:10.875337 5162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.875392 5162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:10.875418 5162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:10.881556 5162 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.66:38275
I20260810 04:20:10.881695 5304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.66:38275 every 8 connection(s)
I20260810 04:20:10.882062 5162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/data/info.pb
I20260810 04:20:10.888087 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.882916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.66:46409 (local address 127.15.216.126:37621)
0810 04:20:10.882976 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:10.882979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:10.883307 (+ 328us) server_negotiation.cc:407] Connection header received
0810 04:20:10.883955 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:10.883957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:10.883972 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:10.884035 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:10.884825 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.885372 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.886494 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.886706 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.886804 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.887756 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:10.887758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:10.887758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:10.887763 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:10.887786 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:10.887791 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:10.887856 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:10.887972 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 04:20:10.888002 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 04:20:10.888303 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:10.882894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:37621 (local address 127.15.216.66:46409)
0810 04:20:10.883134 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:10.883222 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:10.883823 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:10.884123 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:10.884126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:10.884135 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:10.884692 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.884698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.885489 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.885490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.886372 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:10.886375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:10.886819 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:10.886820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:10.886962 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:10.887591 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:10.887591 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:10.887620 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:10.887645 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:10.887858 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:10.887877 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:10.887880 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:10.887971 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 04:20:10.888079 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
I20260810 04:20:10.888939 5305 heartbeater.cc:347] Connected to a master server at 127.15.216.126:37621
I20260810 04:20:10.889034 5305 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:10.889232 5305 heartbeater.cc:511] Master 127.15.216.126:37621 requested a full tablet report, sending...
I20260810 04:20:10.889688 4947 ts_manager.cc:194] Registered new tserver with Master: 004a357b0fac44ad8b2c4affde93ea31 (127.15.216.66:38275)
I20260810 04:20:10.890067 4947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.66:46409
I20260810 04:20:10.891938 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 5162
I20260810 04:20:10.892068 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal/instance
I20260810 04:20:10.892313 16225 external_mini_cluster.cc:1428] Setting key cf726542b3a8679e94ba4908ce99f8ea
I20260810 04:20:10.893260 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.216.67:0
--local_ip_for_outbound_sockets=127.15.216.67
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37621
--builtin_ntp_servers=127.15.216.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 04:20:10.988093 5309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:10.988384 5309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:10.988427 5309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:10.989578 5309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:10.989691 5309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:10.989737 5309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.216.67
I20260810 04:20:10.991112 5309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.216.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.216.67
--webserver_port=0
--tserver_master_addrs=127.15.216.126:37621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.216.67
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:10.991394 5309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:10.991695 5309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:10.996992 5317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.997227 5315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:10.997284 5314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:10.997517 5309 server_base.cc:1034] running on GCE node
I20260810 04:20:10.997750 5309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:10.997962 5309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:10.999104 5309 hybrid_clock.cc:648] HybridClock initialized: now 1786335610999090 us; error 32 us; skew 500 ppm
I20260810 04:20:11.000329 5309 webserver.cc:460] Webserver started at http://127.15.216.67:38313/ using document root <none> and password file <none>
I20260810 04:20:11.000599 5309 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:11.000672 5309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:11.000821 5309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:11.001950 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data/instance:
uuid: "903107a21e2a4d8d89f0f13da87926fd"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "69eac303f9522087316a0ff9338ed537"
server_key_iv: "8beebd3b9ee6de49f6ca735db7faa67f"
server_key_version: "encryptionkey@0"
I20260810 04:20:11.002336 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance:
uuid: "903107a21e2a4d8d89f0f13da87926fd"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "69eac303f9522087316a0ff9338ed537"
server_key_iv: "8beebd3b9ee6de49f6ca735db7faa67f"
server_key_version: "encryptionkey@0"
I20260810 04:20:11.003612 5309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:20:11.004330 5323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:11.004499 5309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:11.004606 5309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal
uuid: "903107a21e2a4d8d89f0f13da87926fd"
format_stamp: "Formatted at 2026-08-10 04:20:10 on dist-test-slave-g350"
server_key: "69eac303f9522087316a0ff9338ed537"
server_key_iv: "8beebd3b9ee6de49f6ca735db7faa67f"
server_key_version: "encryptionkey@0"
I20260810 04:20:11.004681 5309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:11.017560 5309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:11.017975 5309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:11.018134 5309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:11.018388 5309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 04:20:11.018735 5309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 04:20:11.018790 5309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:11.018831 5309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 04:20:11.018860 5309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:11.024976 5309 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.67:36615
I20260810 04:20:11.025032 5445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.67:36615 every 8 connection(s)
I20260810 04:20:11.025354 5309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/data/info.pb
I20260810 04:20:11.029386 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 5309
I20260810 04:20:11.029520 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal/instance
I20260810 04:20:11.029764 16225 external_mini_cluster.cc:1428] Setting key 43c0e929d3780aad1b4025d319a4ff1d
I20260810 04:20:11.030756 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.026135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:52095 (local address 127.15.216.126:37621)
0810 04:20:11.026203 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.026205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.026438 (+ 233us) server_negotiation.cc:407] Connection header received
0810 04:20:11.026970 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.026972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.026986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.027042 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:11.027738 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.028234 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.029213 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.029417 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.029492 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.030472 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:11.030473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:11.030474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:11.030479 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:11.030493 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:11.030503 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:11.030564 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.030650 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.030678 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:20:11.031028 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.026101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:37621 (local address 127.15.216.67:52095)
0810 04:20:11.026293 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:11.026326 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:11.026848 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:11.027122 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:11.027126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:11.027141 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:11.027635 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.027640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.028331 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.028333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.029117 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.029120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.029514 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.029515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.029715 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.030314 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 04:20:11.030315 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:11.030342 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 04:20:11.030364 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:11.030567 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:11.030574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:11.030577 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:11.030650 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 04:20:11.030753 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260810 04:20:11.031476 5446 heartbeater.cc:347] Connected to a master server at 127.15.216.126:37621
I20260810 04:20:11.031562 5446 heartbeater.cc:464] Registering TS with master...
I20260810 04:20:11.031703 5446 heartbeater.cc:511] Master 127.15.216.126:37621 requested a full tablet report, sending...
I20260810 04:20:11.032086 4947 ts_manager.cc:194] Registered new tserver with Master: 903107a21e2a4d8d89f0f13da87926fd (127.15.216.67:36615)
I20260810 04:20:11.032615 4947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.216.67:52095
I20260810 04:20:11.041545 16225 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 04:20:11.046176 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.043080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.15.216.126:37621)
0810 04:20:11.043141 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.043143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.043255 (+ 112us) server_negotiation.cc:407] Connection header received
0810 04:20:11.043356 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.043357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.043370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.043417 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:11.043580 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.044061 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.044848 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.045034 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.045107 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.045899 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:11.045901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:11.045901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:11.045906 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:11.045926 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:11.045930 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:11.045991 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.046055 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.046079 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 04:20:11.047811 4947 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 04:20:11.049368 4947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 04:20:11.056155 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.051911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.15.216.67:36615)
0810 04:20:11.052529 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.052534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.052543 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:11.052582 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.052584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.052600 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.052736 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:11.052847 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.053659 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.055138 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.055676 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.055770 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.055935 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.056016 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.056059 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":88,"threads_started":1}
I20260810 04:20:11.056141 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.051793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.67:36615 (local address 127.0.0.1:51312)
0810 04:20:11.051993 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:11.052049 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:11.052118 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:11.052745 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:11.052746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:11.052748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:11.052790 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.052792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.053752 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.053753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.055039 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.055042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.055785 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.055786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.055891 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.055920 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 04:20:11.055999 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 04:20:11.056069 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":87,"threads_started":1}
I20260810 04:20:11.056425 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.052032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33575 (local address 127.0.0.1:57772)
0810 04:20:11.052222 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:11.052238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:11.052403 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:11.052920 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:11.052921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:11.052923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:11.052967 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.052969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.054243 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.054244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.055129 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.055132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.056005 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.056007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.056069 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.056074 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 04:20:11.056106 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 04:20:11.056148 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260810 04:20:11.056749 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.051799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38275 (local address 127.0.0.1:53368)
0810 04:20:11.052180 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:11.052191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:11.052257 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:11.052491 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:11.052494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:11.052497 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:11.052572 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.052574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.053965 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.053965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.055048 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.055049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.056094 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.056095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.056148 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.056151 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:11.056195 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 04:20:11.056684 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":91,"threads_started":1}
I20260810 04:20:11.057118 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.051983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.15.216.66:38275)
0810 04:20:11.052167 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.052171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.052320 (+ 149us) server_negotiation.cc:407] Connection header received
0810 04:20:11.052377 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.052380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.052398 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.052476 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:11.052658 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.053879 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.055145 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.055879 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.055973 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.056062 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.057001 (+ 939us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.057044 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260810 04:20:11.056973 5375 tablet_service.cc:1467] Processing CreateTablet for tablet 51cd370825e747788408ea03f862b98d (DEFAULT_TABLE table=test-table [id=da816b071f964e8088a025716a3e13ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:11.057402 5375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cd370825e747788408ea03f862b98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:11.057834 5231 tablet_service.cc:1467] Processing CreateTablet for tablet 51cd370825e747788408ea03f862b98d (DEFAULT_TABLE table=test-table [id=da816b071f964e8088a025716a3e13ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:11.058166 5231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cd370825e747788408ea03f862b98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:11.060396 5462 tablet_bootstrap.cc:492] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Bootstrap starting.
I20260810 04:20:11.060488 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.052267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.15.216.65:33575)
0810 04:20:11.052733 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.052736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.052746 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:20:11.052786 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.052787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.052801 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.053038 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:11.053100 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.054138 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.055213 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.055895 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.056199 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.056367 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.056392 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.056431 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":91,"threads_started":1}
I20260810 04:20:11.061133 5463 tablet_bootstrap.cc:492] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Bootstrap starting.
I20260810 04:20:11.061098 5084 tablet_service.cc:1467] Processing CreateTablet for tablet 51cd370825e747788408ea03f862b98d (DEFAULT_TABLE table=test-table [id=da816b071f964e8088a025716a3e13ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 04:20:11.061303 5462 tablet_bootstrap.cc:654] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:11.061443 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cd370825e747788408ea03f862b98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:11.061782 5462 log.cc:824] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:11.061918 5463 tablet_bootstrap.cc:654] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:11.062276 5463 log.cc:824] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:11.062613 5462 tablet_bootstrap.cc:492] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: No bootstrap required, opened a new log
I20260810 04:20:11.062695 5462 ts_tablet_manager.cc:1397] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 04:20:11.063119 5463 tablet_bootstrap.cc:492] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: No bootstrap required, opened a new log
I20260810 04:20:11.063237 5463 ts_tablet_manager.cc:1397] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 04:20:11.064988 5466 tablet_bootstrap.cc:492] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: Bootstrap starting.
I20260810 04:20:11.065285 5463 raft_consensus.cc:348] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.065478 5463 raft_consensus.cc:374] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:11.065515 5463 raft_consensus.cc:729] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004a357b0fac44ad8b2c4affde93ea31, State: Initialized, Role: FOLLOWER
I20260810 04:20:11.065735 5466 tablet_bootstrap.cc:654] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:11.065596 5462 raft_consensus.cc:348] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.065683 5463 consensus_queue.cc:260] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.065804 5462 raft_consensus.cc:374] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:11.065850 5462 raft_consensus.cc:729] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903107a21e2a4d8d89f0f13da87926fd, State: Initialized, Role: FOLLOWER
I20260810 04:20:11.065958 5462 consensus_queue.cc:260] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.065999 5463 ts_tablet_manager.cc:1428] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 04:20:11.066078 5466 log.cc:824] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:11.066118 5305 heartbeater.cc:503] Master 127.15.216.126:37621 was elected leader, sending a full tablet report...
I20260810 04:20:11.066224 5462 ts_tablet_manager.cc:1428] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260810 04:20:11.066375 5446 heartbeater.cc:503] Master 127.15.216.126:37621 was elected leader, sending a full tablet report...
I20260810 04:20:11.067451 5466 tablet_bootstrap.cc:492] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: No bootstrap required, opened a new log
I20260810 04:20:11.067541 5466 ts_tablet_manager.cc:1397] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 04:20:11.069293 5466 raft_consensus.cc:348] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.069435 5466 raft_consensus.cc:374] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:11.069482 5466 raft_consensus.cc:729] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b3dee39218468eacb827c87fb72cbc, State: Initialized, Role: FOLLOWER
I20260810 04:20:11.069618 5466 consensus_queue.cc:260] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.069878 5466 ts_tablet_manager.cc:1428] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 04:20:11.069943 5159 heartbeater.cc:503] Master 127.15.216.126:37621 was elected leader, sending a full tablet report...
W20260810 04:20:11.132838 5307 tablet.cc:2369] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:11.250417 5160 tablet.cc:2369] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 04:20:11.276014 5447 tablet.cc:2369] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 04:20:11.293910 5468 raft_consensus.cc:482] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 04:20:11.294054 5468 raft_consensus.cc:504] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.294446 5468 leader_election.cc:288] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 004a357b0fac44ad8b2c4affde93ea31 (127.15.216.66:38275), 90b3dee39218468eacb827c87fb72cbc (127.15.216.65:33575)
I20260810 04:20:11.298543 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.294631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.66:38275 (local address 127.15.216.67:45555)
0810 04:20:11.294726 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:11.294739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:11.294788 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:11.294993 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:11.294995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:11.294996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:11.295097 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.295099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.296088 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.296089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.297456 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.297459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.298264 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.298265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.298351 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.298355 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 04:20:11.298415 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 04:20:11.298461 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":112}
I20260810 04:20:11.298580 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.294736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:45555 (local address 127.15.216.66:38275)
0810 04:20:11.294789 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.294791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.294803 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:20:11.294884 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.294886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.294900 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.294959 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:11.295189 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.295986 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.297571 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.298167 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.298251 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.298261 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.298433 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.298493 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:20:11.298631 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.294813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.216.67:56687 (local address 127.15.216.65:33575)
0810 04:20:11.294863 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.294865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.294948 (+ 83us) server_negotiation.cc:407] Connection header received
0810 04:20:11.295025 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.295026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.295036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.295081 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 04:20:11.295301 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.296004 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.297571 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.298167 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.298328 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.298335 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.298517 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.298557 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:20:11.298645 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.294681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33575 (local address 127.15.216.67:56687)
0810 04:20:11.294863 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:11.294873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:11.294961 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:11.295101 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:11.295102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:11.295103 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 04:20:11.295168 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.295169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.296088 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.296089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.297476 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:11.297478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.298348 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.298349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:11.298439 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.298442 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 04:20:11.298494 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:20:11.298525 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":56,"thread_start_us":109,"threads_started":1}
I20260810 04:20:11.298974 5108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd370825e747788408ea03f862b98d" candidate_uuid: "903107a21e2a4d8d89f0f13da87926fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b3dee39218468eacb827c87fb72cbc" is_pre_election: true
I20260810 04:20:11.298987 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd370825e747788408ea03f862b98d" candidate_uuid: "903107a21e2a4d8d89f0f13da87926fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004a357b0fac44ad8b2c4affde93ea31" is_pre_election: true
I20260810 04:20:11.299144 5255 raft_consensus.cc:2454] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903107a21e2a4d8d89f0f13da87926fd in term 0.
I20260810 04:20:11.299149 5108 raft_consensus.cc:2454] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 903107a21e2a4d8d89f0f13da87926fd in term 0.
I20260810 04:20:11.299419 5326 leader_election.cc:302] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 903107a21e2a4d8d89f0f13da87926fd, 90b3dee39218468eacb827c87fb72cbc; no voters:
I20260810 04:20:11.299546 5468 raft_consensus.cc:2781] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 04:20:11.299606 5468 raft_consensus.cc:482] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 04:20:11.299629 5468 raft_consensus.cc:3037] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:11.300334 5468 raft_consensus.cc:504] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.300485 5468 leader_election.cc:288] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [CANDIDATE]: Term 1 election: Requested vote from peers 004a357b0fac44ad8b2c4affde93ea31 (127.15.216.66:38275), 90b3dee39218468eacb827c87fb72cbc (127.15.216.65:33575)
I20260810 04:20:11.300690 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd370825e747788408ea03f862b98d" candidate_uuid: "903107a21e2a4d8d89f0f13da87926fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "004a357b0fac44ad8b2c4affde93ea31"
I20260810 04:20:11.300763 5255 raft_consensus.cc:3037] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:11.300835 5108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cd370825e747788408ea03f862b98d" candidate_uuid: "903107a21e2a4d8d89f0f13da87926fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b3dee39218468eacb827c87fb72cbc"
I20260810 04:20:11.300951 5108 raft_consensus.cc:3037] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:11.301680 5255 raft_consensus.cc:2454] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903107a21e2a4d8d89f0f13da87926fd in term 1.
I20260810 04:20:11.301868 5327 leader_election.cc:302] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 004a357b0fac44ad8b2c4affde93ea31, 903107a21e2a4d8d89f0f13da87926fd; no voters:
I20260810 04:20:11.301896 5108 raft_consensus.cc:2454] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 903107a21e2a4d8d89f0f13da87926fd in term 1.
I20260810 04:20:11.301977 5468 raft_consensus.cc:2781] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:11.302170 5468 raft_consensus.cc:686] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 1 LEADER]: Becoming Leader. State: Replica: 903107a21e2a4d8d89f0f13da87926fd, State: Running, Role: LEADER
I20260810 04:20:11.302285 5468 consensus_queue.cc:237] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } }
I20260810 04:20:11.303164 4947 catalog_manager.cc:5495] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 903107a21e2a4d8d89f0f13da87926fd (127.15.216.67). New cstate: current_term: 1 leader_uuid: "903107a21e2a4d8d89f0f13da87926fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "903107a21e2a4d8d89f0f13da87926fd" member_type: VOTER last_known_addr { host: "127.15.216.67" port: 36615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 } health_report { overall_health: UNKNOWN } } }
I20260810 04:20:11.338649 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.336300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.15.216.67:36615)
0810 04:20:11.336361 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:11.336371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:11.336379 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:11.336419 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:11.336420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:11.336440 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:11.336501 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:11.336632 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.337100 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.337909 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:11.338095 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:11.338146 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:11.338287 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:11.338392 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:11.338452 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:11.338554 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:20:11.338570 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:20:11.340689 5108 raft_consensus.cc:1261] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 1 FOLLOWER]: Refusing update from remote peer 903107a21e2a4d8d89f0f13da87926fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:11.340689 5255 raft_consensus.cc:1261] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 1 FOLLOWER]: Refusing update from remote peer 903107a21e2a4d8d89f0f13da87926fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:11.341063 5468 consensus_queue.cc:1035] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "004a357b0fac44ad8b2c4affde93ea31" member_type: VOTER last_known_addr { host: "127.15.216.66" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:11.341292 5472 consensus_queue.cc:1035] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b3dee39218468eacb827c87fb72cbc" member_type: VOTER last_known_addr { host: "127.15.216.65" port: 33575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:11.343712 5476 mvcc.cc:205] Tried to move back new op lower bound from 7316830664048013312 to 7316830663894745088. Current Snapshot: MvccSnapshot[applied={T|T < 7316830664048013312}]
I20260810 04:20:11.344619 5477 mvcc.cc:205] Tried to move back new op lower bound from 7316830664048013312 to 7316830663894745088. Current Snapshot: MvccSnapshot[applied={T|T < 7316830664048013312 or (T in {7316830664048013312})}]
I20260810 04:20:11.346946 4945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:11.347066 4945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 04:20:11.347919 4945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af496bed22a441599d9606041fc47596: Sending DeleteTablet for 3 replicas of tablet 51cd370825e747788408ea03f862b98d
I20260810 04:20:11.348316 5084 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cd370825e747788408ea03f862b98d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:11 UTC) from {username='slave'} at 127.0.0.1:57772
I20260810 04:20:11.348322 5375 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cd370825e747788408ea03f862b98d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:11 UTC) from {username='slave'} at 127.0.0.1:51312
I20260810 04:20:11.348428 5231 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cd370825e747788408ea03f862b98d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 04:20:11 UTC) from {username='slave'} at 127.0.0.1:53368
I20260810 04:20:11.348582 5488 tablet_replica.cc:330] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc: stopping tablet replica
I20260810 04:20:11.348697 5488 raft_consensus.cc:2229] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:11.348834 5488 raft_consensus.cc:2258] T 51cd370825e747788408ea03f862b98d P 90b3dee39218468eacb827c87fb72cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:11.348853 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 5016
I20260810 04:20:11.349001 5489 tablet_replica.cc:330] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: stopping tablet replica
I20260810 04:20:11.349102 5489 raft_consensus.cc:2229] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 1 LEADER]: Raft consensus shutting down.
I20260810 04:20:11.349237 5489 raft_consensus.cc:2258] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:11.350528 5489 ts_tablet_manager.cc:1905] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:11.353122 5489 ts_tablet_manager.cc:1918] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:11.353245 5489 log.cc:1196] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-2/wal/wals/51cd370825e747788408ea03f862b98d
I20260810 04:20:11.353672 5489 ts_tablet_manager.cc:1939] T 51cd370825e747788408ea03f862b98d P 903107a21e2a4d8d89f0f13da87926fd: Deleting consensus metadata
I20260810 04:20:11.354137 5490 tablet_replica.cc:330] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: stopping tablet replica
I20260810 04:20:11.354246 5490 raft_consensus.cc:2229] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 04:20:11.354362 5490 raft_consensus.cc:2258] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 04:20:11.357736 4928 catalog_manager.cc:4841] TS 903107a21e2a4d8d89f0f13da87926fd (127.15.216.67:36615): tablet 51cd370825e747788408ea03f862b98d (table test-table [id=da816b071f964e8088a025716a3e13ab]) successfully deleted
I20260810 04:20:11.359371 5490 ts_tablet_manager.cc:1905] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 04:20:11.361636 5490 ts_tablet_manager.cc:1918] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 04:20:11.361742 5490 log.cc:1196] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Deleting WAL directory at /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786335362448781-16225-0/minicluster-data/ts-1/wal/wals/51cd370825e747788408ea03f862b98d
I20260810 04:20:11.362038 5490 ts_tablet_manager.cc:1939] T 51cd370825e747788408ea03f862b98d P 004a357b0fac44ad8b2c4affde93ea31: Deleting consensus metadata
I20260810 04:20:11.362531 4929 catalog_manager.cc:4841] TS 004a357b0fac44ad8b2c4affde93ea31 (127.15.216.66:38275): tablet 51cd370825e747788408ea03f862b98d (table test-table [id=da816b071f964e8088a025716a3e13ab]) successfully deleted
W20260810 04:20:11.364595 4928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.216.65:33575 (error 108)
I20260810 04:20:11.364826 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 5162
I20260810 04:20:11.366062 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:11.365004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.65:33575 (local address 127.0.0.1:57782)
0810 04:20:11.365976 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:11.366026 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":949}
W20260810 04:20:11.366225 4928 catalog_manager.cc:4584] TS 90b3dee39218468eacb827c87fb72cbc (127.15.216.65:33575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51cd370825e747788408ea03f862b98d: Network error: Client connection negotiation failed: client connection to 127.15.216.65:33575: connect: Connection refused (error 111)
I20260810 04:20:11.372612 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 5309
I20260810 04:20:11.378469 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 4910
2026-08-10T04:20:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (958 ms)
[----------] 6 tests from EncryptionPolicyTest (9283 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:11 dist-test-slave-g350 krb5kdc[5494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:11 dist-test-slave-g350 krb5kdc[5494](info): set up 2 sockets
Aug 10 04:20:11 dist-test-slave-g350 krb5kdc[5494](info): commencing operation
krb5kdc: starting...
W20260810 04:20:12.490670 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:12 dist-test-slave-g350 krb5kdc[5494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:12.620313 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:12.702512 5509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:12.702736 5509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:12.702760 5509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:12.703826 5509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:12.703863 5509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:12.703877 5509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:12.703889 5509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:12.703902 5509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:12.705297 5509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:12.705511 5509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:12.705720 5509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:20:12.710911 5509 server_base.cc:1034] running on GCE node
W20260810 04:20:12.710979 5517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:12.711023 5519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:12.710942 5516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:12.711570 5509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:12.711915 5509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:12.713090 5509 hybrid_clock.cc:648] HybridClock initialized: now 1786335612713075 us; error 36 us; skew 500 ppm
Aug 10 04:20:12 dist-test-slave-g350 krb5kdc[5494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:12.720871 5509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:12.721431 5509 webserver.cc:460] Webserver started at http://127.0.0.1:39907/ using document root <none> and password file <none>
I20260810 04:20:12.721761 5509 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:12.721835 5509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:12.722002 5509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:12.723140 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "1d04eabc44134b1f87ee206412774f1d"
format_stamp: "Formatted at 2026-08-10 04:20:12 on dist-test-slave-g350"
server_key: "10e272f31583cf490a50da59a68a87f7"
server_key_iv: "2deb5ca35509fe57f770a248ba032ac8"
server_key_version: "encryptionkey@0"
I20260810 04:20:12.723543 5509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "1d04eabc44134b1f87ee206412774f1d"
format_stamp: "Formatted at 2026-08-10 04:20:12 on dist-test-slave-g350"
server_key: "10e272f31583cf490a50da59a68a87f7"
server_key_iv: "2deb5ca35509fe57f770a248ba032ac8"
server_key_version: "encryptionkey@0"
I20260810 04:20:12.725051 5509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:12.726032 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:12.726272 5509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:12.726387 5509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "1d04eabc44134b1f87ee206412774f1d"
format_stamp: "Formatted at 2026-08-10 04:20:12 on dist-test-slave-g350"
server_key: "10e272f31583cf490a50da59a68a87f7"
server_key_iv: "2deb5ca35509fe57f770a248ba032ac8"
server_key_version: "encryptionkey@0"
I20260810 04:20:12.726486 5509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:12.753499 5509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:12.755851 5509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:12.756053 5509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:12.760604 5509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36251
I20260810 04:20:12.760641 5591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36251 every 8 connection(s)
I20260810 04:20:12.761049 5509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:12.762053 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:12.764346 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d: Bootstrap starting.
I20260810 04:20:12.764911 5592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:12.765161 5592 log.cc:824] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:12.765918 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d: No bootstrap required, opened a new log
I20260810 04:20:12.766309 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 5509
I20260810 04:20:12.766436 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:12.766649 16225 external_mini_cluster.cc:1428] Setting key 3ac858d93fa9e563207af0738ca0addd
I20260810 04:20:12.767930 5592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d04eabc44134b1f87ee206412774f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260810 04:20:12.768110 5592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:12.768148 5592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d04eabc44134b1f87ee206412774f1d, State: Initialized, Role: FOLLOWER
I20260810 04:20:12.768306 5592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d04eabc44134b1f87ee206412774f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260810 04:20:12.768417 5592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:12.768464 5592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:12.768515 5592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:12.769440 5592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d04eabc44134b1f87ee206412774f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260810 04:20:12.769572 5592 leader_election.cc:302] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d04eabc44134b1f87ee206412774f1d; no voters:
I20260810 04:20:12.769779 5592 leader_election.cc:288] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:12.769809 5597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:12.769932 5597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [term 1 LEADER]: Becoming Leader. State: Replica: 1d04eabc44134b1f87ee206412774f1d, State: Running, Role: LEADER
I20260810 04:20:12.770076 5592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:12.770118 5597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d04eabc44134b1f87ee206412774f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } }
I20260810 04:20:12.770591 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d04eabc44134b1f87ee206412774f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d04eabc44134b1f87ee206412774f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } } }
I20260810 04:20:12.770874 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:12.770794 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d04eabc44134b1f87ee206412774f1d. Latest consensus state: current_term: 1 leader_uuid: "1d04eabc44134b1f87ee206412774f1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d04eabc44134b1f87ee206412774f1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36251 } } }
I20260810 04:20:12.771360 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:12.771435 5607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:12.772756 5607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 04:20:12 dist-test-slave-g350 krb5kdc[5494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:20:12.774784 5607 catalog_manager.cc:1347] Generated new cluster ID: d3f2a9ea2010422aae362da5a1448e1f
I20260810 04:20:12.774847 5607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:12.778977 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:12.767280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33783 (local address 127.0.0.1:36251)
0810 04:20:12.767513 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:12.767521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:12.767548 (+ 27us) server_negotiation.cc:407] Connection header received
0810 04:20:12.767965 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:12.767975 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:12.768044 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:12.768287 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:12.769086 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:12.770302 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:12.771423 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:12.771914 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:12.771999 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:12.775112 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:12.775116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:12.775117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:12.775128 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:12.777142 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:12.777656 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:12.777659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:12.777660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:12.777791 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:12.778060 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:12.778061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:12.778061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:12.778371 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:12.778542 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:12.778670 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 04:20:12.778773 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
I20260810 04:20:12.780088 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:12.789237 5607 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:12.790042 5607 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:12.793715 5596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:12.781564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46431 (local address 127.0.0.1:36251)
0810 04:20:12.781620 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:12.781622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:12.785840 (+ 4218us) server_negotiation.cc:407] Connection header received
0810 04:20:12.785924 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:12.785925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:12.785940 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:12.785999 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:12.786699 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:12.787515 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:12.788549 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:12.788928 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:12.789002 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:12.790556 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:12.790558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:12.790558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:12.790566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:12.792332 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:12.792806 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:12.792808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:12.792809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:12.792876 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:12.793132 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:12.793133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:12.793133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:12.793269 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:12.793378 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:12.793530 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 04:20:12.793624 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:20:12.798224 5607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d04eabc44134b1f87ee206412774f1d: Generated new TSK 0
I20260810 04:20:12.798451 5607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:12.802045 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 5509
W20260810 04:20:12.809656 5492 connection.cc:535] client connection to 127.0.0.1:36251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36251): Connection reset by peer (error 104)
2026-08-10T04:20:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:12 dist-test-slave-g350 krb5kdc[5624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:12 dist-test-slave-g350 krb5kdc[5624](info): set up 2 sockets
Aug 10 04:20:12 dist-test-slave-g350 krb5kdc[5624](info): commencing operation
krb5kdc: starting...
W20260810 04:20:13.883663 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:13 dist-test-slave-g350 krb5kdc[5624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:14.025763 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:14.110532 5638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:14.110767 5638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:14.110793 5638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:14.111871 5638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:14.111917 5638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:14.111935 5638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:14.111949 5638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:14.111963 5638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:14.113392 5638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:14.113682 5638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:14.113889 5638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:14.119494 5644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:14.119494 5646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:14.119524 5638 server_base.cc:1034] running on GCE node
W20260810 04:20:14.119498 5643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:14.120018 5638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:14.120330 5638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:14.121497 5638 hybrid_clock.cc:648] HybridClock initialized: now 1786335614121490 us; error 44 us; skew 500 ppm
Aug 10 04:20:14 dist-test-slave-g350 krb5kdc[5624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:14.124073 5638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:14.124565 5638 webserver.cc:460] Webserver started at http://127.0.0.1:40587/ using document root <none> and password file <none>
I20260810 04:20:14.124783 5638 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:14.124825 5638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:14.124914 5638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:14.126060 5638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "915d77521bc64bde82b90c54973a844c"
format_stamp: "Formatted at 2026-08-10 04:20:14 on dist-test-slave-g350"
server_key: "0e65988faf1ea5008560aaea3687f163"
server_key_iv: "3ff9698ab339e03999cdd8549c58e3c8"
server_key_version: "encryptionkey@0"
I20260810 04:20:14.126375 5638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "915d77521bc64bde82b90c54973a844c"
format_stamp: "Formatted at 2026-08-10 04:20:14 on dist-test-slave-g350"
server_key: "0e65988faf1ea5008560aaea3687f163"
server_key_iv: "3ff9698ab339e03999cdd8549c58e3c8"
server_key_version: "encryptionkey@0"
I20260810 04:20:14.127718 5638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 04:20:14.128539 5654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:14.128768 5638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:14.128880 5638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "915d77521bc64bde82b90c54973a844c"
format_stamp: "Formatted at 2026-08-10 04:20:14 on dist-test-slave-g350"
server_key: "0e65988faf1ea5008560aaea3687f163"
server_key_iv: "3ff9698ab339e03999cdd8549c58e3c8"
server_key_version: "encryptionkey@0"
I20260810 04:20:14.128943 5638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:14.138967 5638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:14.141386 5638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:14.141507 5638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:14.145820 5638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45529
I20260810 04:20:14.145964 5713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45529 every 8 connection(s)
I20260810 04:20:14.146278 5638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:14.147217 5714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:14.149488 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c: Bootstrap starting.
I20260810 04:20:14.150115 5714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:14.150370 5714 log.cc:824] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:14.151037 5714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c: No bootstrap required, opened a new log
I20260810 04:20:14.151726 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 5638
I20260810 04:20:14.151830 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:14.152055 16225 external_mini_cluster.cc:1428] Setting key 244fb2a585348f2aaf4a80c01caddb49
I20260810 04:20:14.153398 5714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915d77521bc64bde82b90c54973a844c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260810 04:20:14.153697 5714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:14.153796 5714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915d77521bc64bde82b90c54973a844c, State: Initialized, Role: FOLLOWER
I20260810 04:20:14.153945 5714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915d77521bc64bde82b90c54973a844c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260810 04:20:14.154034 5714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:14.154057 5714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:14.154122 5714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:14.154688 5714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915d77521bc64bde82b90c54973a844c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260810 04:20:14.154774 5714 leader_election.cc:302] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 915d77521bc64bde82b90c54973a844c; no voters:
I20260810 04:20:14.154947 5714 leader_election.cc:288] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:14.154984 5719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:14.155089 5719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [term 1 LEADER]: Becoming Leader. State: Replica: 915d77521bc64bde82b90c54973a844c, State: Running, Role: LEADER
I20260810 04:20:14.155220 5714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:14.155225 5719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915d77521bc64bde82b90c54973a844c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } }
I20260810 04:20:14.155650 5719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 915d77521bc64bde82b90c54973a844c. Latest consensus state: current_term: 1 leader_uuid: "915d77521bc64bde82b90c54973a844c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915d77521bc64bde82b90c54973a844c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } } }
I20260810 04:20:14.155664 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "915d77521bc64bde82b90c54973a844c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915d77521bc64bde82b90c54973a844c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45529 } } }
I20260810 04:20:14.155896 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:14.155819 5719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:14.156772 5729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:14.157497 5729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:14.158953 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:14.152637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44677 (local address 127.0.0.1:45529)
0810 04:20:14.152878 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:14.152889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:14.152924 (+ 35us) server_negotiation.cc:407] Connection header received
0810 04:20:14.153349 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:14.153358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:14.153453 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:14.153624 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:14.153921 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:14.155475 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:14.156641 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:14.157230 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:14.157359 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:14.158215 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:14.158219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:14.158219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:14.158435 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:14.158473 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:14.158487 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:14.158577 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:14.158694 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 04:20:14.158733 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260810 04:20:14.159086 5729 catalog_manager.cc:1347] Generated new cluster ID: 88f58457e9ad43e5b7dcf590e42f2ff8
I20260810 04:20:14.159129 5729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:14.159937 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:14.164680 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:14.161320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35565 (local address 127.0.0.1:45529)
0810 04:20:14.161359 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:14.161361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:14.161911 (+ 550us) server_negotiation.cc:407] Connection header received
0810 04:20:14.161976 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:14.161977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:14.161991 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:14.162049 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:14.162203 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:14.162700 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:14.163427 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:14.163650 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:14.163728 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:14.164432 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:14.164434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:14.164434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:14.164439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:14.164453 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:14.164457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:14.164510 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:14.164571 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 04:20:14.164584 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:20:14.180691 5729 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:14.181350 5729 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:14.186287 5729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 915d77521bc64bde82b90c54973a844c: Generated new TSK 0
I20260810 04:20:14.186506 5729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:14.191994 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 5638
W20260810 04:20:14.199156 5622 connection.cc:535] client connection to 127.0.0.1:45529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45529): Connection reset by peer (error 104)
2026-08-10T04:20:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:14 dist-test-slave-g350 krb5kdc[5748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:14 dist-test-slave-g350 krb5kdc[5748](info): set up 2 sockets
Aug 10 04:20:14 dist-test-slave-g350 krb5kdc[5748](info): commencing operation
krb5kdc: starting...
W20260810 04:20:15.277947 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:15 dist-test-slave-g350 krb5kdc[5748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:15.406802 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:15.488519 5762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:15.488771 5762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:15.488809 5762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:15.490007 5762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:15.490070 5762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:15.490097 5762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:15.490145 5762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:15.490176 5762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:15.491613 5762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:15.491886 5762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:15.492108 5762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:15.497001 5770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:15.497021 5767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:15.497224 5762 server_base.cc:1034] running on GCE node
W20260810 04:20:15.497021 5768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:15.497543 5762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:15.497864 5762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:15.499055 5762 hybrid_clock.cc:648] HybridClock initialized: now 1786335615499034 us; error 33 us; skew 500 ppm
Aug 10 04:20:15 dist-test-slave-g350 krb5kdc[5748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:15.501634 5762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:15.502187 5762 webserver.cc:460] Webserver started at http://127.0.0.1:44673/ using document root <none> and password file <none>
I20260810 04:20:15.502419 5762 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:15.502491 5762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:15.502625 5762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:15.503665 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "3782afc9052a4fb1b1873699c85b2afa"
format_stamp: "Formatted at 2026-08-10 04:20:15 on dist-test-slave-g350"
server_key: "edcf61b77725f900dc5f3412678984bc"
server_key_iv: "148d680d68de60a30e89760dd472e8ee"
server_key_version: "encryptionkey@0"
I20260810 04:20:15.504041 5762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "3782afc9052a4fb1b1873699c85b2afa"
format_stamp: "Formatted at 2026-08-10 04:20:15 on dist-test-slave-g350"
server_key: "edcf61b77725f900dc5f3412678984bc"
server_key_iv: "148d680d68de60a30e89760dd472e8ee"
server_key_version: "encryptionkey@0"
I20260810 04:20:15.505365 5762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:20:15.506194 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:15.506368 5762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:15.506472 5762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "3782afc9052a4fb1b1873699c85b2afa"
format_stamp: "Formatted at 2026-08-10 04:20:15 on dist-test-slave-g350"
server_key: "edcf61b77725f900dc5f3412678984bc"
server_key_iv: "148d680d68de60a30e89760dd472e8ee"
server_key_version: "encryptionkey@0"
I20260810 04:20:15.506560 5762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:15.509923 5762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:15.512250 5762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:15.512434 5762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:15.516567 5762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34617
I20260810 04:20:15.516601 5836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34617 every 8 connection(s)
I20260810 04:20:15.516940 5762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:15.517840 5837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:15.520027 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa: Bootstrap starting.
I20260810 04:20:15.520599 5837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:15.520884 5837 log.cc:824] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:15.521559 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa: No bootstrap required, opened a new log
I20260810 04:20:15.521908 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 5762
I20260810 04:20:15.522034 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:15.522217 16225 external_mini_cluster.cc:1428] Setting key c7e54b9d5d0fd32af6751e384da3ae96
I20260810 04:20:15.523876 5837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3782afc9052a4fb1b1873699c85b2afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34617 } }
I20260810 04:20:15.524077 5837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:15.524127 5837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3782afc9052a4fb1b1873699c85b2afa, State: Initialized, Role: FOLLOWER
I20260810 04:20:15.524266 5837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3782afc9052a4fb1b1873699c85b2afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34617 } }
I20260810 04:20:15.524381 5837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:15.524434 5837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:15.524489 5837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:15.524709 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:15.522703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58101 (local address 127.0.0.1:34617)
0810 04:20:15.522921 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:15.522929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:15.522965 (+ 36us) server_negotiation.cc:407] Connection header received
0810 04:20:15.523398 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:15.523404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:15.523427 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:15.523541 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:15.524210 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:15.524214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:15.524214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:15.524219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:15.524268 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:15.524281 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:15.524378 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:15.524483 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 04:20:15.524547 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":145,"threads_started":1}
I20260810 04:20:15.525576 5837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3782afc9052a4fb1b1873699c85b2afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34617 } }
I20260810 04:20:15.525691 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:15.525715 5837 leader_election.cc:302] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3782afc9052a4fb1b1873699c85b2afa; no voters:
I20260810 04:20:15.525871 5837 leader_election.cc:288] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:15.525898 5842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:15.526002 5842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [term 1 LEADER]: Becoming Leader. State: Replica: 3782afc9052a4fb1b1873699c85b2afa, State: Running, Role: LEADER
I20260810 04:20:15.526119 5837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:15.526150 5842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3782afc9052a4fb1b1873699c85b2afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34617 } }
I20260810 04:20:15.526484 5842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3782afc9052a4fb1b1873699c85b2afa. Latest consensus state: current_term: 1 leader_uuid: "3782afc9052a4fb1b1873699c85b2afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3782afc9052a4fb1b1873699c85b2afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34617 } } }
I20260810 04:20:15.526585 5842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:15.526494 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3782afc9052a4fb1b1873699c85b2afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3782afc9052a4fb1b1873699c85b2afa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34617 } } }
I20260810 04:20:15.526751 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:15.526805 5852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:15.527343 5852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:15.528577 5841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:15.527284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56883 (local address 127.0.0.1:34617)
0810 04:20:15.527318 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:15.527320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:15.527482 (+ 162us) server_negotiation.cc:407] Connection header received
0810 04:20:15.527590 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:15.527591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:15.527598 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:15.527634 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:15.528348 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:15.528349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:15.528350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:15.528354 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:15.528367 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:15.528371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:15.528400 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:15.528507 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 04:20:15.528521 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:20:15.529132 5852 catalog_manager.cc:1347] Generated new cluster ID: 6188779dc9574eecab6d5d95ca26737a
I20260810 04:20:15.529183 5852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:15.546486 5852 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:15.546658 5852 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:15.561236 5852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3782afc9052a4fb1b1873699c85b2afa: Generated new TSK 0
I20260810 04:20:15.561445 5852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:15.566169 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 5762
2026-08-10T04:20:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:15 dist-test-slave-g350 krb5kdc[5871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:15 dist-test-slave-g350 krb5kdc[5871](info): set up 2 sockets
Aug 10 04:20:15 dist-test-slave-g350 krb5kdc[5871](info): commencing operation
krb5kdc: starting...
W20260810 04:20:16.650637 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:16 dist-test-slave-g350 krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:16.781513 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:16.868646 5887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:16.868854 5887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:16.868877 5887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:16.870043 5887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:16.870087 5887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:16.870102 5887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:16.870115 5887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:16.870153 5887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:16.871632 5887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:16.871884 5887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:16.872109 5887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:16.877323 5892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:16.877331 5895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:16.877530 5893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:16.877729 5887 server_base.cc:1034] running on GCE node
I20260810 04:20:16.878050 5887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:16.878355 5887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:16.879554 5887 hybrid_clock.cc:648] HybridClock initialized: now 1786335616879539 us; error 30 us; skew 500 ppm
Aug 10 04:20:16 dist-test-slave-g350 krb5kdc[5871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:16.882450 5887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:16.882939 5887 webserver.cc:460] Webserver started at http://127.0.0.1:37285/ using document root <none> and password file <none>
I20260810 04:20:16.883129 5887 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:16.883172 5887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:16.883265 5887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:16.884320 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "b61c142ec19b42a59f1b811e894c454b"
format_stamp: "Formatted at 2026-08-10 04:20:16 on dist-test-slave-g350"
server_key: "22d8e98ae636d38f5a69aa35713547a3"
server_key_iv: "8042b9536ed434d455edb15c5503b024"
server_key_version: "encryptionkey@0"
I20260810 04:20:16.884673 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "b61c142ec19b42a59f1b811e894c454b"
format_stamp: "Formatted at 2026-08-10 04:20:16 on dist-test-slave-g350"
server_key: "22d8e98ae636d38f5a69aa35713547a3"
server_key_iv: "8042b9536ed434d455edb15c5503b024"
server_key_version: "encryptionkey@0"
I20260810 04:20:16.886106 5887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:16.886897 5903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:16.887115 5887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:16.887216 5887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "b61c142ec19b42a59f1b811e894c454b"
format_stamp: "Formatted at 2026-08-10 04:20:16 on dist-test-slave-g350"
server_key: "22d8e98ae636d38f5a69aa35713547a3"
server_key_iv: "8042b9536ed434d455edb15c5503b024"
server_key_version: "encryptionkey@0"
I20260810 04:20:16.887295 5887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:16.904089 5887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:16.906963 5887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:16.907137 5887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:16.911394 5887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39593
I20260810 04:20:16.911428 5960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39593 every 8 connection(s)
I20260810 04:20:16.911788 5887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:16.912653 5961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:16.914958 5961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b: Bootstrap starting.
I20260810 04:20:16.915481 5961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:16.915735 5961 log.cc:824] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:16.916394 5961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b: No bootstrap required, opened a new log
I20260810 04:20:16.917471 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 5887
I20260810 04:20:16.917634 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:16.917821 16225 external_mini_cluster.cc:1428] Setting key 08f2c3a0cc1cf9a57043801f5b1f6d89
I20260810 04:20:16.918205 5961 raft_consensus.cc:348] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61c142ec19b42a59f1b811e894c454b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260810 04:20:16.918423 5961 raft_consensus.cc:374] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:16.918479 5961 raft_consensus.cc:729] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61c142ec19b42a59f1b811e894c454b, State: Initialized, Role: FOLLOWER
I20260810 04:20:16.918668 5961 consensus_queue.cc:260] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61c142ec19b42a59f1b811e894c454b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260810 04:20:16.918757 5961 raft_consensus.cc:388] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:16.918790 5961 raft_consensus.cc:482] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:16.918839 5961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:16.920017 5961 raft_consensus.cc:504] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61c142ec19b42a59f1b811e894c454b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260810 04:20:16.920113 5961 leader_election.cc:302] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b61c142ec19b42a59f1b811e894c454b; no voters:
I20260810 04:20:16.920310 5961 leader_election.cc:288] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:16.920339 5967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:16.920466 5967 raft_consensus.cc:686] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [term 1 LEADER]: Becoming Leader. State: Replica: b61c142ec19b42a59f1b811e894c454b, State: Running, Role: LEADER
I20260810 04:20:16.920575 5961 sys_catalog.cc:564] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:16.920727 5967 consensus_queue.cc:237] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61c142ec19b42a59f1b811e894c454b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } }
I20260810 04:20:16.921159 5968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b61c142ec19b42a59f1b811e894c454b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61c142ec19b42a59f1b811e894c454b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } } }
I20260810 04:20:16.921253 5968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:16.921772 5977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:16.922263 5977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:16.924856 5977 catalog_manager.cc:1347] Generated new cluster ID: 0d871d43ccbe4ca89ab3c87233229f68
I20260810 04:20:16.924921 5977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:20:16 dist-test-slave-g350 krb5kdc[5871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:20:16.922286 5970 sys_catalog.cc:455] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b61c142ec19b42a59f1b811e894c454b. Latest consensus state: current_term: 1 leader_uuid: "b61c142ec19b42a59f1b811e894c454b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b61c142ec19b42a59f1b811e894c454b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39593 } } }
I20260810 04:20:16.925649 5970 sys_catalog.cc:458] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:16.929762 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:16.918406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42049 (local address 127.0.0.1:39593)
0810 04:20:16.918591 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:16.918600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:16.918632 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:20:16.918744 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:16.918752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:16.918809 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:16.918953 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:16.919958 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:16.921243 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:16.922456 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:16.923022 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:16.923133 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:16.925933 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:16.925937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:16.925938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:16.925954 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:16.927955 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:16.928350 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:16.928353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:16.928354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:16.928478 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:16.928689 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:16.928690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:16.928690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:16.929044 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:16.929210 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:16.929306 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 04:20:16.929486 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260810 04:20:16.931025 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:16.939607 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:16.932449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44069 (local address 127.0.0.1:39593)
0810 04:20:16.932489 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:16.932491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:16.932622 (+ 131us) server_negotiation.cc:407] Connection header received
0810 04:20:16.932655 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:16.932656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:16.932667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:16.932722 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:16.933312 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:16.934060 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:16.935066 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:16.935388 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:16.935458 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:16.936811 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:16.936813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:16.936813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:16.936820 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:16.938424 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:16.938803 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:16.938804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:16.938805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:16.938862 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:16.939076 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:16.939076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:16.939077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:16.939236 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:16.939335 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:16.939434 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:20:16.939508 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:20:16.950098 5977 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:16.950637 5977 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:16.956792 5977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b61c142ec19b42a59f1b811e894c454b: Generated new TSK 0
I20260810 04:20:16.957013 5977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:16.963644 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 5887
2026-08-10T04:20:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:17 dist-test-slave-g350 krb5kdc[5994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:17 dist-test-slave-g350 krb5kdc[5994](info): set up 2 sockets
Aug 10 04:20:17 dist-test-slave-g350 krb5kdc[5994](info): commencing operation
krb5kdc: starting...
W20260810 04:20:18.043859 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:18 dist-test-slave-g350 krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:18.172399 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:18.256974 6009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:18.257277 6009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:18.257325 6009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:18.258757 6009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:18.258846 6009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:18.258878 6009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:18.258912 6009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:18.258951 6009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:18.260675 6009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:18.261027 6009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:18.261297 6009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:18.266942 6018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:18.266959 6014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:18.266963 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:18.267009 6009 server_base.cc:1034] running on GCE node
I20260810 04:20:18.267503 6009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:18.267817 6009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:18.268987 6009 hybrid_clock.cc:648] HybridClock initialized: now 1786335618268942 us; error 57 us; skew 500 ppm
Aug 10 04:20:18 dist-test-slave-g350 krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:18.275102 6009 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:18.275669 6009 webserver.cc:460] Webserver started at http://127.0.0.1:33777/ using document root <none> and password file <none>
I20260810 04:20:18.275966 6009 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:18.276050 6009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:18.276216 6009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:18.277383 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "e92f95c9914344ea98172ccd23da6436"
format_stamp: "Formatted at 2026-08-10 04:20:18 on dist-test-slave-g350"
server_key: "71cfe51c7d853a1ed0703452507b50b0"
server_key_iv: "bb42426dcd22ecff565c8a5abae66cb8"
server_key_version: "encryptionkey@0"
I20260810 04:20:18.277812 6009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "e92f95c9914344ea98172ccd23da6436"
format_stamp: "Formatted at 2026-08-10 04:20:18 on dist-test-slave-g350"
server_key: "71cfe51c7d853a1ed0703452507b50b0"
server_key_iv: "bb42426dcd22ecff565c8a5abae66cb8"
server_key_version: "encryptionkey@0"
I20260810 04:20:18.279412 6009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:18.280292 6025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:18.280555 6009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:18.280685 6009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e92f95c9914344ea98172ccd23da6436"
format_stamp: "Formatted at 2026-08-10 04:20:18 on dist-test-slave-g350"
server_key: "71cfe51c7d853a1ed0703452507b50b0"
server_key_iv: "bb42426dcd22ecff565c8a5abae66cb8"
server_key_version: "encryptionkey@0"
I20260810 04:20:18.280771 6009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:18.293037 6009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:18.295321 6009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:18.295507 6009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:18.300493 6009 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37817
I20260810 04:20:18.300547 6083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37817 every 8 connection(s)
I20260810 04:20:18.300956 6009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:18.301939 6084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:18.304291 6084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436: Bootstrap starting.
I20260810 04:20:18.304931 6084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:18.305243 6084 log.cc:824] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:18.305995 6084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436: No bootstrap required, opened a new log
I20260810 04:20:18.307657 6084 raft_consensus.cc:348] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92f95c9914344ea98172ccd23da6436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37817 } }
I20260810 04:20:18.307811 6084 raft_consensus.cc:374] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:18.307834 6084 raft_consensus.cc:729] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92f95c9914344ea98172ccd23da6436, State: Initialized, Role: FOLLOWER
I20260810 04:20:18.308040 6084 consensus_queue.cc:260] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92f95c9914344ea98172ccd23da6436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37817 } }
I20260810 04:20:18.308146 6084 raft_consensus.cc:388] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:18.308171 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6009
I20260810 04:20:18.308183 6084 raft_consensus.cc:482] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:18.308247 6084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:18.308288 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:18.308503 16225 external_mini_cluster.cc:1428] Setting key 5be5cf3657af1034fa5a1e787a517a9a
I20260810 04:20:18.308954 6084 raft_consensus.cc:504] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92f95c9914344ea98172ccd23da6436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37817 } }
I20260810 04:20:18.309129 6084 leader_election.cc:302] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92f95c9914344ea98172ccd23da6436; no voters:
I20260810 04:20:18.309350 6084 leader_election.cc:288] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:18.309422 6089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:18.309576 6089 raft_consensus.cc:686] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [term 1 LEADER]: Becoming Leader. State: Replica: e92f95c9914344ea98172ccd23da6436, State: Running, Role: LEADER
I20260810 04:20:18.309796 6089 consensus_queue.cc:237] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92f95c9914344ea98172ccd23da6436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37817 } }
I20260810 04:20:18.309644 6084 sys_catalog.cc:564] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:18.310309 6091 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e92f95c9914344ea98172ccd23da6436. Latest consensus state: current_term: 1 leader_uuid: "e92f95c9914344ea98172ccd23da6436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92f95c9914344ea98172ccd23da6436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37817 } } }
I20260810 04:20:18.310461 6091 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:18.310870 6090 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e92f95c9914344ea98172ccd23da6436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92f95c9914344ea98172ccd23da6436" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37817 } } }
I20260810 04:20:18.310992 6090 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:18.311151 6094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:18.311636 6094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:18.314424 6094 catalog_manager.cc:1347] Generated new cluster ID: 12eefd93465f483da772c2a55c169279
I20260810 04:20:18.314476 6094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:18.315002 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:18.309021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44033 (local address 127.0.0.1:37817)
0810 04:20:18.309241 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:18.309252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:18.309289 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:20:18.309454 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:18.309462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:18.309530 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:18.309762 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:18.309986 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:18.311744 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:18.312652 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:18.313238 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:18.313382 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:18.314219 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:18.314223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:18.314223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:18.314463 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:18.314503 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:18.314520 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:18.314593 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:18.314700 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 04:20:18.314769 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260810 04:20:18.316026 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:18.325345 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:18.317744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41757 (local address 127.0.0.1:37817)
0810 04:20:18.317803 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:18.317805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:18.321815 (+ 4010us) server_negotiation.cc:407] Connection header received
0810 04:20:18.321886 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:18.321887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:18.321898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:18.321957 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:18.322105 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:18.322837 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:18.323974 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:18.324265 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:18.324322 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:18.325058 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:18.325059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:18.325059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:18.325078 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:18.325094 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:18.325098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:18.325148 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:18.325237 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 04:20:18.325264 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 04:20:18.340823 6094 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:18.341717 6094 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:18.350104 6094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e92f95c9914344ea98172ccd23da6436: Generated new TSK 0
I20260810 04:20:18.350314 6094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:18.357264 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6009
2026-08-10T04:20:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:18 dist-test-slave-g350 krb5kdc[6119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:18 dist-test-slave-g350 krb5kdc[6119](info): set up 2 sockets
Aug 10 04:20:18 dist-test-slave-g350 krb5kdc[6119](info): commencing operation
krb5kdc: starting...
W20260810 04:20:19.458391 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:19 dist-test-slave-g350 krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:19.585505 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:19.668228 6133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:19.668466 6133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:19.668504 6133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:19.669667 6133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:19.669735 6133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:19.669760 6133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:19.669792 6133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:19.669821 6133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:19.671319 6133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:19.671598 6133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:19.671819 6133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:19.677017 6139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:19.677008 6141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:19.677008 6143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:19.677258 6133 server_base.cc:1034] running on GCE node
I20260810 04:20:19.677577 6133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:19.677925 6133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:19.679095 6133 hybrid_clock.cc:648] HybridClock initialized: now 1786335619679045 us; error 40 us; skew 500 ppm
Aug 10 04:20:19 dist-test-slave-g350 krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:19.681741 6133 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:19.682329 6133 webserver.cc:460] Webserver started at http://127.0.0.1:46285/ using document root <none> and password file <none>
I20260810 04:20:19.682619 6133 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:19.682690 6133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:19.682845 6133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:19.684058 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "4a881d91f5b0401b8b3f564e992ce8d2"
format_stamp: "Formatted at 2026-08-10 04:20:19 on dist-test-slave-g350"
server_key: "7971c7bdc55809b19125bbcf81aa460d"
server_key_iv: "aef320485bf345e6d222894ac884e81d"
server_key_version: "encryptionkey@0"
I20260810 04:20:19.684448 6133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "4a881d91f5b0401b8b3f564e992ce8d2"
format_stamp: "Formatted at 2026-08-10 04:20:19 on dist-test-slave-g350"
server_key: "7971c7bdc55809b19125bbcf81aa460d"
server_key_iv: "aef320485bf345e6d222894ac884e81d"
server_key_version: "encryptionkey@0"
I20260810 04:20:19.685845 6133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:20:19.686592 6150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:19.686770 6133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:19.686867 6133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "4a881d91f5b0401b8b3f564e992ce8d2"
format_stamp: "Formatted at 2026-08-10 04:20:19 on dist-test-slave-g350"
server_key: "7971c7bdc55809b19125bbcf81aa460d"
server_key_iv: "aef320485bf345e6d222894ac884e81d"
server_key_version: "encryptionkey@0"
I20260810 04:20:19.686944 6133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:19.690235 6133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:19.692046 6133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:19.692201 6133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:19.696379 6133 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35653
I20260810 04:20:19.696414 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35653 every 8 connection(s)
I20260810 04:20:19.696727 6133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:19.697546 6209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:19.699770 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2: Bootstrap starting.
I20260810 04:20:19.700380 6209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:19.700567 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6133
I20260810 04:20:19.700700 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:19.700685 6209 log.cc:824] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:19.700912 16225 external_mini_cluster.cc:1428] Setting key 535bed97ef72239bbb0f91e5ab806c27
I20260810 04:20:19.701582 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2: No bootstrap required, opened a new log
I20260810 04:20:19.703382 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:19.701422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38669 (local address 127.0.0.1:35653)
0810 04:20:19.701635 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:19.701644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:19.701676 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:20:19.702128 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:19.702135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:19.702158 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:19.702301 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:19.702985 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:19.702989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:19.702989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:19.702994 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:19.703040 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:19.703050 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:19.703141 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:19.703179 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 04:20:19.703217 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260810 04:20:19.704409 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:19.704391 6209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260810 04:20:19.704571 6209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:19.704618 6209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a881d91f5b0401b8b3f564e992ce8d2, State: Initialized, Role: FOLLOWER
I20260810 04:20:19.704739 6209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260810 04:20:19.704846 6209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:19.704892 6209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:19.704941 6209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:19.705523 6209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260810 04:20:19.705673 6209 leader_election.cc:302] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a881d91f5b0401b8b3f564e992ce8d2; no voters:
I20260810 04:20:19.705884 6209 leader_election.cc:288] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:19.705906 6219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:19.706022 6219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4a881d91f5b0401b8b3f564e992ce8d2, State: Running, Role: LEADER
I20260810 04:20:19.706103 6219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } }
I20260810 04:20:19.706143 6209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:19.706420 6219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } } }
I20260810 04:20:19.706494 6219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:19.706617 6219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a881d91f5b0401b8b3f564e992ce8d2. Latest consensus state: current_term: 1 leader_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a881d91f5b0401b8b3f564e992ce8d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35653 } } }
I20260810 04:20:19.706691 6219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:19.706832 6227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:19.707378 6227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:19.708773 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:19.706041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54649 (local address 127.0.0.1:35653)
0810 04:20:19.706075 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:19.706077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:19.706156 (+ 79us) server_negotiation.cc:407] Connection header received
0810 04:20:19.706249 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:19.706249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:19.706254 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:19.706310 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:19.708511 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:19.708513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:19.708513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:19.708517 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:19.708532 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:19.708536 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:19.708599 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:19.708679 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 04:20:19.708696 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 04:20:19.709173 6227 catalog_manager.cc:1347] Generated new cluster ID: 651a5be9912843619fd37ce85eae59a4
I20260810 04:20:19.709213 6227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:19.716611 6227 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:19.716771 6227 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:19.725695 6227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a881d91f5b0401b8b3f564e992ce8d2: Generated new TSK 0
I20260810 04:20:19.725894 6227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:19.730468 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6133
2026-08-10T04:20:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:19 dist-test-slave-g350 krb5kdc[6244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:19 dist-test-slave-g350 krb5kdc[6244](info): set up 2 sockets
Aug 10 04:20:19 dist-test-slave-g350 krb5kdc[6244](info): commencing operation
krb5kdc: starting...
W20260810 04:20:20.815084 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:20 dist-test-slave-g350 krb5kdc[6244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:20.944556 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:21.026712 6259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:21.026968 6259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:21.027007 6259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:21.028134 6259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:21.028208 6259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:21.028232 6259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:21.028258 6259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:21.028288 6259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:21.029804 6259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:21.030092 6259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:21.030323 6259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:20:21.035534 6259 server_base.cc:1034] running on GCE node
W20260810 04:20:21.035490 6268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:21.035491 6266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:21.035475 6265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:21.036028 6259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:21.036279 6259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:21.037432 6259 hybrid_clock.cc:648] HybridClock initialized: now 1786335621037421 us; error 33 us; skew 500 ppm
Aug 10 04:20:21 dist-test-slave-g350 krb5kdc[6244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:21.040016 6259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:21.040442 6259 webserver.cc:460] Webserver started at http://127.0.0.1:39969/ using document root <none> and password file <none>
I20260810 04:20:21.040683 6259 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:21.040727 6259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:21.040860 6259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:21.041963 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "5d0327261d3547ce875bf324714ed9de"
format_stamp: "Formatted at 2026-08-10 04:20:21 on dist-test-slave-g350"
server_key: "124587e0076cfa8d99bb52799fb9d8fe"
server_key_iv: "f60254b1ca3fcfe5a5edce32df468e4f"
server_key_version: "encryptionkey@0"
I20260810 04:20:21.042309 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "5d0327261d3547ce875bf324714ed9de"
format_stamp: "Formatted at 2026-08-10 04:20:21 on dist-test-slave-g350"
server_key: "124587e0076cfa8d99bb52799fb9d8fe"
server_key_iv: "f60254b1ca3fcfe5a5edce32df468e4f"
server_key_version: "encryptionkey@0"
I20260810 04:20:21.043617 6259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:21.044440 6276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:21.044627 6259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:21.044740 6259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "5d0327261d3547ce875bf324714ed9de"
format_stamp: "Formatted at 2026-08-10 04:20:21 on dist-test-slave-g350"
server_key: "124587e0076cfa8d99bb52799fb9d8fe"
server_key_iv: "f60254b1ca3fcfe5a5edce32df468e4f"
server_key_version: "encryptionkey@0"
I20260810 04:20:21.044833 6259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:21.058804 6259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:21.060660 6259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:21.060830 6259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:21.064842 6259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39053
I20260810 04:20:21.064903 6334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39053 every 8 connection(s)
I20260810 04:20:21.065248 6259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:21.066148 6336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:21.068382 6336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de: Bootstrap starting.
I20260810 04:20:21.068966 6336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:21.069255 6336 log.cc:824] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:21.069783 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6259
I20260810 04:20:21.069909 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:21.070112 16225 external_mini_cluster.cc:1428] Setting key 386fadca2d46d0a7b3917853b593f2d4
I20260810 04:20:21.070171 6336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de: No bootstrap required, opened a new log
I20260810 04:20:21.072470 6336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0327261d3547ce875bf324714ed9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260810 04:20:21.072642 6336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:21.072682 6336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d0327261d3547ce875bf324714ed9de, State: Initialized, Role: FOLLOWER
I20260810 04:20:21.072824 6336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0327261d3547ce875bf324714ed9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260810 04:20:21.072925 6336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:21.072958 6336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:21.073010 6336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:21.073748 6336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0327261d3547ce875bf324714ed9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260810 04:20:21.073855 6336 leader_election.cc:302] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d0327261d3547ce875bf324714ed9de; no voters:
I20260810 04:20:21.074015 6336 leader_election.cc:288] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:21.074086 6341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:21.074231 6336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:21.074230 6341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [term 1 LEADER]: Becoming Leader. State: Replica: 5d0327261d3547ce875bf324714ed9de, State: Running, Role: LEADER
I20260810 04:20:21.074474 6341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0327261d3547ce875bf324714ed9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } }
I20260810 04:20:21.074860 6343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d0327261d3547ce875bf324714ed9de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0327261d3547ce875bf324714ed9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } } }
I20260810 04:20:21.074954 6343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:21.075112 6342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d0327261d3547ce875bf324714ed9de. Latest consensus state: current_term: 1 leader_uuid: "5d0327261d3547ce875bf324714ed9de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d0327261d3547ce875bf324714ed9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39053 } } }
I20260810 04:20:21.075183 6342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:21.075305 6352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:21.075821 6352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:21.077489 6352 catalog_manager.cc:1347] Generated new cluster ID: 6d69dcc0020340c5b828970ad17247ed
I20260810 04:20:21.077548 6352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:20:21 dist-test-slave-g350 krb5kdc[6244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:20:21.082370 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:21.070781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:40119 (local address 127.0.0.1:39053)
0810 04:20:21.071003 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:21.071012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:21.071043 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:20:21.071657 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:21.071666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:21.071737 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:21.071881 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:21.072507 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:21.074238 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:21.075472 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:21.076197 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:21.076292 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:21.078989 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:21.078993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:21.078993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:21.079005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:21.080706 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:21.081122 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:21.081127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:21.081132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:21.081216 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:21.081454 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:21.081456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:21.081456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:21.081797 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:21.082003 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:21.082093 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 04:20:21.082193 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260810 04:20:21.083437 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:21.091850 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:21.084853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:41477 (local address 127.0.0.1:39053)
0810 04:20:21.084890 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:21.084892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:21.085876 (+ 984us) server_negotiation.cc:407] Connection header received
0810 04:20:21.085938 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:21.085939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:21.085950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:21.085999 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:21.086517 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:21.087017 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:21.087735 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:21.087965 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:21.088043 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:21.089209 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:21.089210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:21.089210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:21.089217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:21.090645 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:21.091026 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:21.091027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:21.091027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:21.091086 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:21.091309 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:21.091310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:21.091311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:21.091421 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:21.091531 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:21.091657 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 04:20:21.091721 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:20:21.109118 6352 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:21.109735 6352 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:21.113822 6352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d0327261d3547ce875bf324714ed9de: Generated new TSK 0
I20260810 04:20:21.114023 6352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:21.123090 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6259
W20260810 04:20:21.130272 6242 connection.cc:535] client connection to 127.0.0.1:39053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39053): Connection reset by peer (error 104)
2026-08-10T04:20:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:21 dist-test-slave-g350 krb5kdc[6370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:21 dist-test-slave-g350 krb5kdc[6370](info): set up 2 sockets
Aug 10 04:20:21 dist-test-slave-g350 krb5kdc[6370](info): commencing operation
krb5kdc: starting...
W20260810 04:20:22.223820 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:22 dist-test-slave-g350 krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:22.348583 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:22.430706 6384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:22.430959 6384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:22.430996 6384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:22.432127 6384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:22.432206 6384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:22.432231 6384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:22.432255 6384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:22.432284 6384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:22.433830 6384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:22.434114 6384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:22.434351 6384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:22.439330 6392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:22.439481 6384 server_base.cc:1034] running on GCE node
W20260810 04:20:22.439320 6390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:22.439321 6389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:22.439869 6384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:22.440124 6384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:22.441291 6384 hybrid_clock.cc:648] HybridClock initialized: now 1786335622441282 us; error 39 us; skew 500 ppm
Aug 10 04:20:22 dist-test-slave-g350 krb5kdc[6370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:22.443935 6384 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:22.444432 6384 webserver.cc:460] Webserver started at http://127.0.0.1:38227/ using document root <none> and password file <none>
I20260810 04:20:22.444666 6384 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:22.444736 6384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:22.444880 6384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:22.446060 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "5d6b60bd58d04a9db7ba804ead5690d2"
format_stamp: "Formatted at 2026-08-10 04:20:22 on dist-test-slave-g350"
server_key: "8d1b655048e85a6b15a875865c9f677b"
server_key_iv: "91fc948d12e2573eadfb567d0ca7f237"
server_key_version: "encryptionkey@0"
I20260810 04:20:22.446445 6384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "5d6b60bd58d04a9db7ba804ead5690d2"
format_stamp: "Formatted at 2026-08-10 04:20:22 on dist-test-slave-g350"
server_key: "8d1b655048e85a6b15a875865c9f677b"
server_key_iv: "91fc948d12e2573eadfb567d0ca7f237"
server_key_version: "encryptionkey@0"
I20260810 04:20:22.447788 6384 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:22.448575 6400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:22.448771 6384 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:22.448897 6384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "5d6b60bd58d04a9db7ba804ead5690d2"
format_stamp: "Formatted at 2026-08-10 04:20:22 on dist-test-slave-g350"
server_key: "8d1b655048e85a6b15a875865c9f677b"
server_key_iv: "91fc948d12e2573eadfb567d0ca7f237"
server_key_version: "encryptionkey@0"
I20260810 04:20:22.448976 6384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:22.462100 6384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:22.463937 6384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:22.464107 6384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:22.468340 6384 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46173
I20260810 04:20:22.468375 6457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46173 every 8 connection(s)
I20260810 04:20:22.468717 6384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:22.469559 6458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:22.471817 6458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2: Bootstrap starting.
I20260810 04:20:22.472326 6458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:22.472579 6458 log.cc:824] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:22.473212 6458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2: No bootstrap required, opened a new log
I20260810 04:20:22.473721 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6384
I20260810 04:20:22.473867 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:22.474058 16225 external_mini_cluster.cc:1428] Setting key a7314f7a62c270413f825fac76b54d51
I20260810 04:20:22.475426 6458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260810 04:20:22.475559 6458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:22.475581 6458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6b60bd58d04a9db7ba804ead5690d2, State: Initialized, Role: FOLLOWER
I20260810 04:20:22.475709 6458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260810 04:20:22.475807 6458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:22.475841 6458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:22.475960 6458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:22.476529 6458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260810 04:20:22.476619 6458 leader_election.cc:302] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d6b60bd58d04a9db7ba804ead5690d2; no voters:
I20260810 04:20:22.476792 6458 leader_election.cc:288] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:22.476828 6463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:22.477003 6463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [term 1 LEADER]: Becoming Leader. State: Replica: 5d6b60bd58d04a9db7ba804ead5690d2, State: Running, Role: LEADER
I20260810 04:20:22.477061 6458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:22.477154 6463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260810 04:20:22.477770 6464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } } }
I20260810 04:20:22.477885 6464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:22.477783 6465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d6b60bd58d04a9db7ba804ead5690d2. Latest consensus state: current_term: 1 leader_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b60bd58d04a9db7ba804ead5690d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } } }
I20260810 04:20:22.477967 6465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:22.478101 6474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:22.478582 6474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:22.480222 6474 catalog_manager.cc:1347] Generated new cluster ID: e33364b7837c436ab66d4a36d8336b8a
I20260810 04:20:22.480274 6474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:22.480706 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:22.474592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:54033 (local address 127.0.0.1:46173)
0810 04:20:22.474784 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:22.474792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:22.474817 (+ 25us) server_negotiation.cc:407] Connection header received
0810 04:20:22.475582 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:22.475592 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:22.475662 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:22.475831 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:22.476095 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:22.477243 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:22.478556 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:22.479127 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:22.479247 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:22.480034 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:22.480038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:22.480039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:22.480246 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:22.480283 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:22.480298 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:22.480384 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:22.480483 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 04:20:22.480510 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260810 04:20:22.481700 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:22.489085 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:22.483103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:38769 (local address 127.0.0.1:46173)
0810 04:20:22.483133 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:22.483135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:22.485814 (+ 2679us) server_negotiation.cc:407] Connection header received
0810 04:20:22.485889 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:22.485890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:22.485900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:22.485958 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:22.486094 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:22.486768 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:22.487772 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:22.488062 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:22.488120 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:22.488841 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:22.488842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:22.488843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:22.488847 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:22.488861 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:22.488865 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:22.488920 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:22.489002 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 04:20:22.489012 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 04:20:22.513322 6474 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:22.514110 6474 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:22.532922 6474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d6b60bd58d04a9db7ba804ead5690d2: Generated new TSK 0
I20260810 04:20:22.533147 6474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:22.535267 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6384
W20260810 04:20:22.542769 6368 connection.cc:535] client connection to 127.0.0.1:46173 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46173): Connection reset by peer (error 104)
2026-08-10T04:20:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:22 dist-test-slave-g350 krb5kdc[6493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:22 dist-test-slave-g350 krb5kdc[6493](info): set up 2 sockets
Aug 10 04:20:22 dist-test-slave-g350 krb5kdc[6493](info): commencing operation
krb5kdc: starting...
W20260810 04:20:23.610625 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:23 dist-test-slave-g350 krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:23.736887 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:23.820972 6509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:23.821190 6509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:23.821221 6509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:23.822455 6509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:23.822510 6509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:23.822525 6509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:23.822538 6509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:23.822551 6509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:23.824123 6509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:23.824402 6509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:23.824641 6509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:23.829746 6516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:23.829756 6518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:23.829849 6509 server_base.cc:1034] running on GCE node
W20260810 04:20:23.829756 6515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:23.830320 6509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:23.830665 6509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:23.831828 6509 hybrid_clock.cc:648] HybridClock initialized: now 1786335623831822 us; error 35 us; skew 500 ppm
Aug 10 04:20:23 dist-test-slave-g350 krb5kdc[6493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:23.838315 6509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:23.838749 6509 webserver.cc:460] Webserver started at http://127.0.0.1:40093/ using document root <none> and password file <none>
I20260810 04:20:23.839006 6509 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:23.839075 6509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:23.839224 6509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:23.840288 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "ecbd69555d0d4ea6a295872b43789279"
format_stamp: "Formatted at 2026-08-10 04:20:23 on dist-test-slave-g350"
server_key: "bf35a61d0586e8ed0647a78d6e274473"
server_key_iv: "2ba9d881b54f4cec9f72436d11257f47"
server_key_version: "encryptionkey@0"
I20260810 04:20:23.840648 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "ecbd69555d0d4ea6a295872b43789279"
format_stamp: "Formatted at 2026-08-10 04:20:23 on dist-test-slave-g350"
server_key: "bf35a61d0586e8ed0647a78d6e274473"
server_key_iv: "2ba9d881b54f4cec9f72436d11257f47"
server_key_version: "encryptionkey@0"
I20260810 04:20:23.842006 6509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:23.842823 6526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:23.843061 6509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:23.843180 6509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "ecbd69555d0d4ea6a295872b43789279"
format_stamp: "Formatted at 2026-08-10 04:20:23 on dist-test-slave-g350"
server_key: "bf35a61d0586e8ed0647a78d6e274473"
server_key_iv: "2ba9d881b54f4cec9f72436d11257f47"
server_key_version: "encryptionkey@0"
I20260810 04:20:23.843255 6509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:23.846601 6509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:23.848431 6509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:23.848580 6509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:23.852623 6509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33583
I20260810 04:20:23.852655 6582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33583 every 8 connection(s)
I20260810 04:20:23.852977 6509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:23.853916 6583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:23.856454 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279: Bootstrap starting.
I20260810 04:20:23.857030 6583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:23.857324 6583 log.cc:824] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:23.858050 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279: No bootstrap required, opened a new log
I20260810 04:20:23.859654 6583 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbd69555d0d4ea6a295872b43789279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33583 } }
I20260810 04:20:23.859786 6583 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:23.859808 6583 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbd69555d0d4ea6a295872b43789279, State: Initialized, Role: FOLLOWER
I20260810 04:20:23.859975 6583 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbd69555d0d4ea6a295872b43789279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33583 } }
I20260810 04:20:23.860056 6583 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:23.860103 6583 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:23.860172 6583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:23.860683 6583 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbd69555d0d4ea6a295872b43789279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33583 } }
I20260810 04:20:23.860773 6583 leader_election.cc:302] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecbd69555d0d4ea6a295872b43789279; no voters:
I20260810 04:20:23.860950 6583 leader_election.cc:288] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:23.861006 6586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:23.861205 6583 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:23.861313 6586 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [term 1 LEADER]: Becoming Leader. State: Replica: ecbd69555d0d4ea6a295872b43789279, State: Running, Role: LEADER
I20260810 04:20:23.861423 6586 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbd69555d0d4ea6a295872b43789279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33583 } }
I20260810 04:20:23.862071 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6509
I20260810 04:20:23.862068 6589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbd69555d0d4ea6a295872b43789279. Latest consensus state: current_term: 1 leader_uuid: "ecbd69555d0d4ea6a295872b43789279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbd69555d0d4ea6a295872b43789279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33583 } } }
I20260810 04:20:23.862169 6589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:23.862203 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:23.862425 16225 external_mini_cluster.cc:1428] Setting key 951f8c372facc2c72c6d8da7440d6e59
I20260810 04:20:23.862534 6588 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecbd69555d0d4ea6a295872b43789279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbd69555d0d4ea6a295872b43789279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33583 } } }
I20260810 04:20:23.862623 6588 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:23.863229 6598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:23.863838 6598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:23.865006 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:23.863009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:56445 (local address 127.0.0.1:33583)
0810 04:20:23.863288 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:23.863297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:23.863334 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:20:23.863449 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:23.863456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:23.863487 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:23.863652 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:23.864369 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:23.864372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:23.864372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:23.864378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:23.864428 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:23.864444 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:23.864540 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:23.864637 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 04:20:23.864795 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":70,"threads_started":1}
I20260810 04:20:23.865420 6598 catalog_manager.cc:1347] Generated new cluster ID: 3a042ac7570e45c9b9e939c47020e3b6
I20260810 04:20:23.865473 6598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:23.869685 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:23.872537 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:23.871348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:60887 (local address 127.0.0.1:33583)
0810 04:20:23.871388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:23.871390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:23.871531 (+ 141us) server_negotiation.cc:407] Connection header received
0810 04:20:23.871585 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:23.871586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:23.871592 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:23.871649 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:23.872287 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:23.872288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:23.872288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:23.872292 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:23.872309 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:23.872314 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:23.872386 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:23.872444 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 04:20:23.872457 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:20:23.877494 6598 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:23.877666 6598 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:23.883911 6598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecbd69555d0d4ea6a295872b43789279: Generated new TSK 0
I20260810 04:20:23.884099 6598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:23.886684 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6509
2026-08-10T04:20:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:23 dist-test-slave-g350 krb5kdc[6618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:23 dist-test-slave-g350 krb5kdc[6618](info): set up 2 sockets
Aug 10 04:20:23 dist-test-slave-g350 krb5kdc[6618](info): commencing operation
krb5kdc: starting...
W20260810 04:20:24.974207 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:25 dist-test-slave-g350 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:25.098251 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:25.180466 6631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:25.180677 6631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:25.180702 6631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:25.181869 6631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:25.181911 6631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:25.181927 6631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:25.181967 6631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:25.181999 6631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:25.183522 6631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:25.183771 6631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:25.184005 6631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:25.189064 6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:25.189072 6636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:25.189280 6631 server_base.cc:1034] running on GCE node
W20260810 04:20:25.189075 6637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:25.189649 6631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:25.189960 6631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:25.191128 6631 hybrid_clock.cc:648] HybridClock initialized: now 1786335625191112 us; error 44 us; skew 500 ppm
Aug 10 04:20:25 dist-test-slave-g350 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:25.193558 6631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:25.194074 6631 webserver.cc:460] Webserver started at http://127.0.0.1:36595/ using document root <none> and password file <none>
I20260810 04:20:25.194319 6631 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:25.194383 6631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:25.194525 6631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:25.195565 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "477bf0ebe43d46748d34fdfd53b7fb34"
format_stamp: "Formatted at 2026-08-10 04:20:25 on dist-test-slave-g350"
server_key: "4f9d654f9d11bb21a503b0fc576415d8"
server_key_iv: "7c2ac58ff141dc5585ee197ea73cefa1"
server_key_version: "encryptionkey@0"
I20260810 04:20:25.195945 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "477bf0ebe43d46748d34fdfd53b7fb34"
format_stamp: "Formatted at 2026-08-10 04:20:25 on dist-test-slave-g350"
server_key: "4f9d654f9d11bb21a503b0fc576415d8"
server_key_iv: "7c2ac58ff141dc5585ee197ea73cefa1"
server_key_version: "encryptionkey@0"
I20260810 04:20:25.197250 6631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:20:25.198041 6647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:25.198211 6631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:25.198318 6631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "477bf0ebe43d46748d34fdfd53b7fb34"
format_stamp: "Formatted at 2026-08-10 04:20:25 on dist-test-slave-g350"
server_key: "4f9d654f9d11bb21a503b0fc576415d8"
server_key_iv: "7c2ac58ff141dc5585ee197ea73cefa1"
server_key_version: "encryptionkey@0"
I20260810 04:20:25.198392 6631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:25.207269 6631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:25.209028 6631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:25.209187 6631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:25.213303 6631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45267
I20260810 04:20:25.213343 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45267 every 8 connection(s)
I20260810 04:20:25.213701 6631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:25.214529 6705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:25.216804 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34: Bootstrap starting.
I20260810 04:20:25.217386 6705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:25.217731 6705 log.cc:824] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:25.218406 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34: No bootstrap required, opened a new log
I20260810 04:20:25.219988 6705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260810 04:20:25.220122 6705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:25.220166 6705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 477bf0ebe43d46748d34fdfd53b7fb34, State: Initialized, Role: FOLLOWER
I20260810 04:20:25.220314 6705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260810 04:20:25.220397 6705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:25.220445 6705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:25.220495 6705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:25.221035 6705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260810 04:20:25.221125 6705 leader_election.cc:302] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 477bf0ebe43d46748d34fdfd53b7fb34; no voters:
I20260810 04:20:25.221297 6705 leader_election.cc:288] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:25.221339 6709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:25.221534 6709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [term 1 LEADER]: Becoming Leader. State: Replica: 477bf0ebe43d46748d34fdfd53b7fb34, State: Running, Role: LEADER
I20260810 04:20:25.221565 6705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:25.221698 6709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } }
I20260810 04:20:25.222136 6711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 477bf0ebe43d46748d34fdfd53b7fb34. Latest consensus state: current_term: 1 leader_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } } }
I20260810 04:20:25.222265 6710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "477bf0ebe43d46748d34fdfd53b7fb34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45267 } } }
I20260810 04:20:25.222227 6711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:25.222342 6710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:25.222529 6718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:25.223214 6718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:25.223575 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6631
I20260810 04:20:25.223694 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:25.223891 16225 external_mini_cluster.cc:1428] Setting key 65b74f65b73b910b8f299ad67d4e3ff2
I20260810 04:20:25.225332 6718 catalog_manager.cc:1347] Generated new cluster ID: 5552d0b1bac54a5bbe2bbf497ec4860e
I20260810 04:20:25.225386 6718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:25.234103 6718 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:25.235231 6718 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:25.238296 6718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 477bf0ebe43d46748d34fdfd53b7fb34: Generated new TSK 0
I20260810 04:20:25.238464 6718 catalog_manager.cc:1514] Initializing in-progress tserver states...
Aug 10 04:20:25 dist-test-slave-g350 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 04:20:25.244614 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:25.224360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:55619 (local address 127.0.0.1:45267)
0810 04:20:25.224536 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:25.224544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:25.233850 (+ 9306us) server_negotiation.cc:407] Connection header received
0810 04:20:25.234029 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:25.234036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:25.234098 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:25.234231 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:25.235331 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:25.236761 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:25.237938 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:25.238563 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:25.238664 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:25.241058 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:25.241062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:25.241062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:25.241075 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:25.243001 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:25.243434 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:25.243437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:25.243437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:25.243546 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:25.243764 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:25.243765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:25.243765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:25.244034 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:25.244195 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:25.244271 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 04:20:25.244387 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260810 04:20:25.245756 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:25.253875 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:25.247147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:33991 (local address 127.0.0.1:45267)
0810 04:20:25.247188 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:25.247190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:25.247291 (+ 101us) server_negotiation.cc:407] Connection header received
0810 04:20:25.247333 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:25.247340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:25.247350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:25.247415 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:25.247902 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:25.248690 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:25.249505 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:25.249841 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:25.249913 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:25.251115 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:25.251116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:25.251117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:25.251123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:25.252608 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:25.253015 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:25.253016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:25.253016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:25.253078 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:25.253345 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:25.253346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:25.253346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:25.253469 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:25.253582 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:25.253720 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 04:20:25.253786 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 04:20:25.255843 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6631
W20260810 04:20:25.262826 6616 connection.cc:535] client connection to 127.0.0.1:45267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45267): Connection reset by peer (error 104)
2026-08-10T04:20:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:25 dist-test-slave-g350 krb5kdc[6739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:25 dist-test-slave-g350 krb5kdc[6739](info): set up 2 sockets
Aug 10 04:20:25 dist-test-slave-g350 krb5kdc[6739](info): commencing operation
krb5kdc: starting...
W20260810 04:20:26.335841 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:26 dist-test-slave-g350 krb5kdc[6739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:26.460789 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:26.543469 6753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:26.543679 6753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:26.543702 6753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:26.544780 6753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:26.544813 6753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:26.544828 6753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:26.544842 6753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:26.544853 6753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:26.546291 6753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:26.546496 6753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:26.546659 6753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:26.551693 6761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:26.551693 6760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:26.551688 6763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:26.552066 6753 server_base.cc:1034] running on GCE node
I20260810 04:20:26.552318 6753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:26.552604 6753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:26.553764 6753 hybrid_clock.cc:648] HybridClock initialized: now 1786335626553757 us; error 34 us; skew 500 ppm
Aug 10 04:20:26 dist-test-slave-g350 krb5kdc[6739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:26.556393 6753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:26.556823 6753 webserver.cc:460] Webserver started at http://127.0.0.1:39761/ using document root <none> and password file <none>
I20260810 04:20:26.557066 6753 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:26.557134 6753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:26.557272 6753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:26.558388 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "e04c70686d9b459b93071cf2351f9016"
format_stamp: "Formatted at 2026-08-10 04:20:26 on dist-test-slave-g350"
server_key: "6ba36be376e3a3a5d64db7b2a1e3ae06"
server_key_iv: "c8ae5fdcc19e86fb8e8ba610e5ce9564"
server_key_version: "encryptionkey@0"
I20260810 04:20:26.558771 6753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "e04c70686d9b459b93071cf2351f9016"
format_stamp: "Formatted at 2026-08-10 04:20:26 on dist-test-slave-g350"
server_key: "6ba36be376e3a3a5d64db7b2a1e3ae06"
server_key_iv: "c8ae5fdcc19e86fb8e8ba610e5ce9564"
server_key_version: "encryptionkey@0"
I20260810 04:20:26.560150 6753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:20:26.560916 6770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:26.561240 6753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:26.561385 6753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e04c70686d9b459b93071cf2351f9016"
format_stamp: "Formatted at 2026-08-10 04:20:26 on dist-test-slave-g350"
server_key: "6ba36be376e3a3a5d64db7b2a1e3ae06"
server_key_iv: "c8ae5fdcc19e86fb8e8ba610e5ce9564"
server_key_version: "encryptionkey@0"
I20260810 04:20:26.561470 6753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:26.582705 6753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:26.585145 6753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:26.585333 6753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:26.589617 6753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43825
I20260810 04:20:26.589699 6825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43825 every 8 connection(s)
I20260810 04:20:26.590070 6753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:26.591017 6826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:26.593319 6826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016: Bootstrap starting.
I20260810 04:20:26.593922 6826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:26.594225 6826 log.cc:824] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:26.594950 6826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016: No bootstrap required, opened a new log
I20260810 04:20:26.596192 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6753
I20260810 04:20:26.596328 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:26.596530 16225 external_mini_cluster.cc:1428] Setting key 418941c95cc9898ffc679d988bc9842c
I20260810 04:20:26.596729 6826 raft_consensus.cc:348] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04c70686d9b459b93071cf2351f9016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260810 04:20:26.596928 6826 raft_consensus.cc:374] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:26.597035 6826 raft_consensus.cc:729] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04c70686d9b459b93071cf2351f9016, State: Initialized, Role: FOLLOWER
I20260810 04:20:26.597252 6826 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04c70686d9b459b93071cf2351f9016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260810 04:20:26.597357 6826 raft_consensus.cc:388] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:26.597393 6826 raft_consensus.cc:482] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:26.597440 6826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:26.598677 6826 raft_consensus.cc:504] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04c70686d9b459b93071cf2351f9016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260810 04:20:26.598789 6826 leader_election.cc:302] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04c70686d9b459b93071cf2351f9016; no voters:
I20260810 04:20:26.598953 6826 leader_election.cc:288] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:26.599010 6832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:26.599176 6826 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:26.599195 6832 raft_consensus.cc:686] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [term 1 LEADER]: Becoming Leader. State: Replica: e04c70686d9b459b93071cf2351f9016, State: Running, Role: LEADER
I20260810 04:20:26.599287 6832 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04c70686d9b459b93071cf2351f9016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } }
I20260810 04:20:26.599785 6832 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04c70686d9b459b93071cf2351f9016. Latest consensus state: current_term: 1 leader_uuid: "e04c70686d9b459b93071cf2351f9016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04c70686d9b459b93071cf2351f9016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } } }
I20260810 04:20:26.599793 6833 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e04c70686d9b459b93071cf2351f9016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04c70686d9b459b93071cf2351f9016" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43825 } } }
I20260810 04:20:26.599879 6832 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:26.599895 6833 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:26.600359 6842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:26.600910 6842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:26.602555 6842 catalog_manager.cc:1347] Generated new cluster ID: 769ffc6e7b48430f82620da8b0d8ccff
I20260810 04:20:26.602609 6842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:26.604050 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:26.597095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:46195 (local address 127.0.0.1:43825)
0810 04:20:26.597315 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:26.597324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:26.597370 (+ 46us) server_negotiation.cc:407] Connection header received
0810 04:20:26.597485 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:26.597493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:26.597553 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:26.597717 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:26.597890 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:26.600466 (+ 2576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:26.601712 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:26.602321 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:26.602444 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:26.603342 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:26.603346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:26.603346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:26.603565 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:26.603606 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:26.603621 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:26.603709 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:26.603810 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 04:20:26.603838 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":115,"threads_started":1}
I20260810 04:20:26.605166 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:26.610124 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:26.606595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:33671 (local address 127.0.0.1:43825)
0810 04:20:26.606636 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:26.606638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:26.606773 (+ 135us) server_negotiation.cc:407] Connection header received
0810 04:20:26.606834 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:26.606835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:26.606844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:26.606900 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:26.607027 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:26.607726 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:26.608741 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:26.609039 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:26.609093 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:26.609874 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:26.609875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:26.609876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:26.609880 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:26.609895 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:26.609899 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:26.609957 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:26.610045 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 04:20:26.610058 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:20:26.611508 6842 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:26.612048 6842 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:26.617718 6842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e04c70686d9b459b93071cf2351f9016: Generated new TSK 0
I20260810 04:20:26.617893 6842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:26.623663 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6753
W20260810 04:20:26.630941 6737 connection.cc:535] client connection to 127.0.0.1:43825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43825): Connection reset by peer (error 104)
2026-08-10T04:20:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:26 dist-test-slave-g350 krb5kdc[6861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:26 dist-test-slave-g350 krb5kdc[6861](info): set up 2 sockets
Aug 10 04:20:26 dist-test-slave-g350 krb5kdc[6861](info): commencing operation
krb5kdc: starting...
W20260810 04:20:27.711663 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:27 dist-test-slave-g350 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 04:20:27.840200 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:27.926040 6875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:27.926286 6875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:27.926324 6875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:27.927460 6875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:27.927529 6875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:27.927556 6875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:27.927584 6875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:27.927614 6875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:27.929087 6875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:27.929364 6875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:27.929579 6875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:27.934634 6880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:27.934645 6883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:27.934645 6881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:27.935312 6875 server_base.cc:1034] running on GCE node
I20260810 04:20:27.935562 6875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:27.935798 6875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:27.936969 6875 hybrid_clock.cc:648] HybridClock initialized: now 1786335627936953 us; error 40 us; skew 500 ppm
Aug 10 04:20:27 dist-test-slave-g350 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:27.939958 6875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 04:20:27.940609 6875 webserver.cc:460] Webserver started at http://127.0.0.1:35423/ using document root <none> and password file <none>
I20260810 04:20:27.940901 6875 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:27.941013 6875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:27.941190 6875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:27.942806 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "9e43c4e21826495d93ba71eb85865d4c"
format_stamp: "Formatted at 2026-08-10 04:20:27 on dist-test-slave-g350"
server_key: "3f5ecaf913b63f1d812b9660b8254f35"
server_key_iv: "bb4e1493f5b077ac07279dffd8fa1112"
server_key_version: "encryptionkey@0"
I20260810 04:20:27.943336 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "9e43c4e21826495d93ba71eb85865d4c"
format_stamp: "Formatted at 2026-08-10 04:20:27 on dist-test-slave-g350"
server_key: "3f5ecaf913b63f1d812b9660b8254f35"
server_key_iv: "bb4e1493f5b077ac07279dffd8fa1112"
server_key_version: "encryptionkey@0"
I20260810 04:20:27.945199 6875 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 04:20:27.947418 6890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:27.947597 6875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260810 04:20:27.947685 6875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "9e43c4e21826495d93ba71eb85865d4c"
format_stamp: "Formatted at 2026-08-10 04:20:27 on dist-test-slave-g350"
server_key: "3f5ecaf913b63f1d812b9660b8254f35"
server_key_iv: "bb4e1493f5b077ac07279dffd8fa1112"
server_key_version: "encryptionkey@0"
I20260810 04:20:27.947786 6875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:27.952484 6875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:27.954810 6875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:27.954977 6875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:27.959023 6875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33255
I20260810 04:20:27.959118 6946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33255 every 8 connection(s)
I20260810 04:20:27.959419 6875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:27.960366 6947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:27.962898 6947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c: Bootstrap starting.
I20260810 04:20:27.963537 6947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:27.963872 6947 log.cc:824] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:27.964597 6947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c: No bootstrap required, opened a new log
I20260810 04:20:27.966028 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6875
I20260810 04:20:27.966118 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:27.966310 16225 external_mini_cluster.cc:1428] Setting key 1574e0d3399c1537ab01bc4a920f651f
I20260810 04:20:27.966625 6947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e43c4e21826495d93ba71eb85865d4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260810 04:20:27.966846 6947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:27.966907 6947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e43c4e21826495d93ba71eb85865d4c, State: Initialized, Role: FOLLOWER
I20260810 04:20:27.967080 6947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e43c4e21826495d93ba71eb85865d4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260810 04:20:27.967190 6947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:27.967229 6947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:27.967286 6947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:27.968125 6947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e43c4e21826495d93ba71eb85865d4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260810 04:20:27.968255 6947 leader_election.cc:302] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e43c4e21826495d93ba71eb85865d4c; no voters:
I20260810 04:20:27.968442 6947 leader_election.cc:288] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:27.968472 6952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:27.968616 6952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [term 1 LEADER]: Becoming Leader. State: Replica: 9e43c4e21826495d93ba71eb85865d4c, State: Running, Role: LEADER
I20260810 04:20:27.968709 6952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e43c4e21826495d93ba71eb85865d4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } }
I20260810 04:20:27.968758 6947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:27.969062 6953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e43c4e21826495d93ba71eb85865d4c. Latest consensus state: current_term: 1 leader_uuid: "9e43c4e21826495d93ba71eb85865d4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e43c4e21826495d93ba71eb85865d4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } } }
I20260810 04:20:27.969156 6953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:27.969259 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:27.966813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:58835 (local address 127.0.0.1:33255)
0810 04:20:27.967013 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:27.967021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:27.967077 (+ 56us) server_negotiation.cc:407] Connection header received
0810 04:20:27.967299 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:27.967306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:27.967349 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:27.967481 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:27.968301 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:27.968305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:27.968306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:27.968310 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:27.968407 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:27.968421 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:27.968511 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:27.968855 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 04:20:27.968912 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260810 04:20:27.969342 6954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e43c4e21826495d93ba71eb85865d4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e43c4e21826495d93ba71eb85865d4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33255 } } }
I20260810 04:20:27.969465 6954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:27.969864 6957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:27.970407 6957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:27.970459 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:27.972146 6957 catalog_manager.cc:1347] Generated new cluster ID: 7b145e553ffd443ca6e5fe2b33ef552c
I20260810 04:20:27.972203 6957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:27.975104 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:27.973877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.171:44133 (local address 127.0.0.1:33255)
0810 04:20:27.973912 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:27.973914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:27.974061 (+ 147us) server_negotiation.cc:407] Connection header received
0810 04:20:27.974171 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:27.974172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:27.974192 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:27.974252 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:27.974872 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:27.974873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:27.974873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:27.974877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:27.974894 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:27.974898 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:27.974956 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:27.975016 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 04:20:27.975029 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:20:27.980436 6957 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:27.980603 6957 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:27.983685 6957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e43c4e21826495d93ba71eb85865d4c: Generated new TSK 0
I20260810 04:20:27.983851 6957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:27.990633 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6875
2026-08-10T04:20:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:28 dist-test-slave-g350 krb5kdc[6981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:28 dist-test-slave-g350 krb5kdc[6981](info): set up 2 sockets
Aug 10 04:20:28 dist-test-slave-g350 krb5kdc[6981](info): commencing operation
krb5kdc: starting...
W20260810 04:20:29.114462 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:29 dist-test-slave-g350 krb5kdc[6981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:29.240175 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:35787
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:35787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:29.323462 6995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:29.323658 6995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:29.323683 6995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:29.324841 6995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:29.324883 6995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:29.324898 6995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:29.324911 6995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:29.324923 6995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:29.326514 6995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:35787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:29.326756 6995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:29.326978 6995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:29.331862 7001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:29.331954 7004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:29.331940 6995 server_base.cc:1034] running on GCE node
W20260810 04:20:29.331874 7000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:29.332423 6995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:29.332713 6995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:29.333866 6995 hybrid_clock.cc:648] HybridClock initialized: now 1786335629333858 us; error 43 us; skew 500 ppm
Aug 10 04:20:29 dist-test-slave-g350 krb5kdc[6981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335629, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:29.339707 6995 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:29.340101 6995 webserver.cc:460] Webserver started at http://127.15.216.126:42293/ using document root <none> and password file <none>
I20260810 04:20:29.340306 6995 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:29.340346 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:29.340435 6995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:29.341416 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "83ae9b8334554610bed66ddde4246d08"
format_stamp: "Formatted at 2026-08-10 04:20:29 on dist-test-slave-g350"
server_key: "51e1587c28e4b83021a3bb7bd834da35"
server_key_iv: "cebc72847b8af47ac619206b39f18b9c"
server_key_version: "encryptionkey@0"
I20260810 04:20:29.341782 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "83ae9b8334554610bed66ddde4246d08"
format_stamp: "Formatted at 2026-08-10 04:20:29 on dist-test-slave-g350"
server_key: "51e1587c28e4b83021a3bb7bd834da35"
server_key_iv: "cebc72847b8af47ac619206b39f18b9c"
server_key_version: "encryptionkey@0"
I20260810 04:20:29.343039 6995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:29.343708 7011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:29.343899 6995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:20:29.343984 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "83ae9b8334554610bed66ddde4246d08"
format_stamp: "Formatted at 2026-08-10 04:20:29 on dist-test-slave-g350"
server_key: "51e1587c28e4b83021a3bb7bd834da35"
server_key_iv: "cebc72847b8af47ac619206b39f18b9c"
server_key_version: "encryptionkey@0"
I20260810 04:20:29.344043 6995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:29.357019 6995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:29.358980 6995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:29.359094 6995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:29.363000 6995 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:35787
I20260810 04:20:29.363049 7067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:35787 every 8 connection(s)
I20260810 04:20:29.363372 6995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:29.364230 7068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:29.365803 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 6995
I20260810 04:20:29.365933 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:29.366147 16225 external_mini_cluster.cc:1428] Setting key 7bcb725602ce921a0b899151f21ef01f
I20260810 04:20:29.366863 7068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08: Bootstrap starting.
I20260810 04:20:29.367549 7068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:29.367851 7068 log.cc:824] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:29.368634 7068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08: No bootstrap required, opened a new log
I20260810 04:20:29.370968 7068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ae9b8334554610bed66ddde4246d08" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 35787 } }
I20260810 04:20:29.371132 7068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:29.371167 7068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ae9b8334554610bed66ddde4246d08, State: Initialized, Role: FOLLOWER
I20260810 04:20:29.371292 7068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ae9b8334554610bed66ddde4246d08" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 35787 } }
I20260810 04:20:29.371399 7068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:29.371435 7068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:29.371532 7068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:29.372197 7068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ae9b8334554610bed66ddde4246d08" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 35787 } }
I20260810 04:20:29.372319 7068 leader_election.cc:302] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83ae9b8334554610bed66ddde4246d08; no voters:
I20260810 04:20:29.372519 7068 leader_election.cc:288] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:29.372540 7073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:29.372655 7073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [term 1 LEADER]: Becoming Leader. State: Replica: 83ae9b8334554610bed66ddde4246d08, State: Running, Role: LEADER
I20260810 04:20:29.372738 7073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ae9b8334554610bed66ddde4246d08" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 35787 } }
I20260810 04:20:29.372778 7068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:29.373051 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83ae9b8334554610bed66ddde4246d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ae9b8334554610bed66ddde4246d08" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 35787 } } }
I20260810 04:20:29.373148 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:29.373304 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83ae9b8334554610bed66ddde4246d08. Latest consensus state: current_term: 1 leader_uuid: "83ae9b8334554610bed66ddde4246d08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ae9b8334554610bed66ddde4246d08" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 35787 } } }
I20260810 04:20:29.373391 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08 [sys.catalog]: This master's current role is: LEADER
Aug 10 04:20:29 dist-test-slave-g350 krb5kdc[6981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:29.373770 7079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:29.374310 7079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:29.377456 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:29.366650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50705 (local address 127.15.216.126:35787)
0810 04:20:29.366817 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:29.366824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:29.366848 (+ 24us) server_negotiation.cc:407] Connection header received
0810 04:20:29.367368 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:29.367376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:29.367437 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:29.367585 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:29.368578 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:29.369976 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:29.370686 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:29.371293 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:29.371410 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:29.374262 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:29.374267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:29.374267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:29.374279 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:29.375922 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:29.376319 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:29.376322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:29.376323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:29.376420 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:29.376644 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:29.376645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:29.376646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:29.376879 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:29.377040 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:29.377169 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 04:20:29.377269 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260810 04:20:29.378487 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:29.382606 7079 catalog_manager.cc:1347] Generated new cluster ID: 6d1d7e1139d44494b388266e46bf5582
I20260810 04:20:29.382663 7079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:29.386276 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:29.379688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50851 (local address 127.15.216.126:35787)
0810 04:20:29.379723 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:29.379724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:29.379812 (+ 88us) server_negotiation.cc:407] Connection header received
0810 04:20:29.379893 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:29.379894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:29.379901 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:29.379948 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:29.380560 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:29.381264 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:29.382274 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:29.382512 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:29.382581 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:29.383808 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:29.383809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:29.383810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:29.383817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:29.385079 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:29.385448 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:29.385449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:29.385449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:29.385509 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:29.385773 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:29.385774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:29.385774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:29.385882 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:29.385999 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:29.386124 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 04:20:29.386184 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:20:29.388833 7079 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:29.389333 7079 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:29.398307 7079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83ae9b8334554610bed66ddde4246d08: Generated new TSK 0
I20260810 04:20:29.398494 7079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:29.404075 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 6995
W20260810 04:20:29.411504 6979 connection.cc:535] client connection to 127.15.216.126:35787 recv error: Network error: failed to read from TLS socket (remote: 127.15.216.126:35787): Connection reset by peer (error 104)
2026-08-10T04:20:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:29 dist-test-slave-g350 krb5kdc[7100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:29 dist-test-slave-g350 krb5kdc[7100](info): set up 2 sockets
Aug 10 04:20:29 dist-test-slave-g350 krb5kdc[7100](info): commencing operation
krb5kdc: starting...
W20260810 04:20:30.518496 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:30 dist-test-slave-g350 krb5kdc[7100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:30.642140 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36849
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:36849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:30.724648 7115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:30.724926 7115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:30.724953 7115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:30.726164 7115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:30.726207 7115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:30.726220 7115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:30.726254 7115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:30.726281 7115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:30.727767 7115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:36849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:36849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:30.728092 7115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:30.728404 7115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:30.733542 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:30.733542 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:30.733548 7125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:30.733560 7115 server_base.cc:1034] running on GCE node
I20260810 04:20:30.734179 7115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:30.734474 7115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:30.735646 7115 hybrid_clock.cc:648] HybridClock initialized: now 1786335630735634 us; error 44 us; skew 500 ppm
Aug 10 04:20:30 dist-test-slave-g350 krb5kdc[7100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335630, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:30.738300 7115 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:30.738807 7115 webserver.cc:460] Webserver started at http://127.15.216.126:41787/ using document root <none> and password file <none>
I20260810 04:20:30.739035 7115 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:30.739104 7115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:30.739284 7115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:30.740360 7115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "77ec26727fd74a9d9aa52f04e3c25017"
format_stamp: "Formatted at 2026-08-10 04:20:30 on dist-test-slave-g350"
server_key: "2e76e146e2f196b6563979c308984784"
server_key_iv: "fcf06471007dc996fedcbfdb8e71bdf4"
server_key_version: "encryptionkey@0"
I20260810 04:20:30.740726 7115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "77ec26727fd74a9d9aa52f04e3c25017"
format_stamp: "Formatted at 2026-08-10 04:20:30 on dist-test-slave-g350"
server_key: "2e76e146e2f196b6563979c308984784"
server_key_iv: "fcf06471007dc996fedcbfdb8e71bdf4"
server_key_version: "encryptionkey@0"
I20260810 04:20:30.742427 7115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:30.743217 7136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:30.743414 7115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:30.743526 7115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "77ec26727fd74a9d9aa52f04e3c25017"
format_stamp: "Formatted at 2026-08-10 04:20:30 on dist-test-slave-g350"
server_key: "2e76e146e2f196b6563979c308984784"
server_key_iv: "fcf06471007dc996fedcbfdb8e71bdf4"
server_key_version: "encryptionkey@0"
I20260810 04:20:30.743599 7115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:30.769683 7115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:30.772141 7115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:30.772315 7115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:30.776744 7115 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:36849
I20260810 04:20:30.776794 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:36849 every 8 connection(s)
I20260810 04:20:30.777195 7115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:30.777468 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7115
I20260810 04:20:30.777619 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:30.777807 16225 external_mini_cluster.cc:1428] Setting key 045ccb6cc8dbbc9c7c1353e922b26dae
I20260810 04:20:30.778184 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:30.781023 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017: Bootstrap starting.
I20260810 04:20:30.781680 7190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:30.781985 7190 log.cc:824] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:30.782876 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017: No bootstrap required, opened a new log
I20260810 04:20:30.783555 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:30.778346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55811 (local address 127.15.216.126:36849)
0810 04:20:30.778549 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:30.778559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:30.778591 (+ 32us) server_negotiation.cc:407] Connection header received
0810 04:20:30.779269 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:30.779276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:30.779324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:30.779475 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:30.779697 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:30.780586 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:30.781429 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:30.781946 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:30.782066 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:30.782838 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:30.782842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:30.782842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:30.783044 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:30.783075 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:30.783086 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:30.783154 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:30.783340 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 04:20:30.783376 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260810 04:20:30.784616 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:30.785478 7190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec26727fd74a9d9aa52f04e3c25017" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36849 } }
I20260810 04:20:30.785727 7190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:30.785804 7190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ec26727fd74a9d9aa52f04e3c25017, State: Initialized, Role: FOLLOWER
I20260810 04:20:30.785959 7190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec26727fd74a9d9aa52f04e3c25017" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36849 } }
I20260810 04:20:30.786072 7190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:30.786182 7190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:30.786254 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:30.786890 7190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec26727fd74a9d9aa52f04e3c25017" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36849 } }
I20260810 04:20:30.787019 7190 leader_election.cc:302] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77ec26727fd74a9d9aa52f04e3c25017; no voters:
I20260810 04:20:30.787220 7190 leader_election.cc:288] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:30.787288 7202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:30.787452 7202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [term 1 LEADER]: Becoming Leader. State: Replica: 77ec26727fd74a9d9aa52f04e3c25017, State: Running, Role: LEADER
I20260810 04:20:30.787485 7190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:30.787617 7202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec26727fd74a9d9aa52f04e3c25017" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36849 } }
I20260810 04:20:30.787931 7202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77ec26727fd74a9d9aa52f04e3c25017. Latest consensus state: current_term: 1 leader_uuid: "77ec26727fd74a9d9aa52f04e3c25017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec26727fd74a9d9aa52f04e3c25017" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36849 } } }
I20260810 04:20:30.788144 7202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:30.788082 7203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77ec26727fd74a9d9aa52f04e3c25017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77ec26727fd74a9d9aa52f04e3c25017" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 36849 } } }
I20260810 04:20:30.788414 7203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:30.788836 7212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:30.789685 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:30.786331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38897 (local address 127.15.216.126:36849)
0810 04:20:30.786409 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:30.786411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:30.786419 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:30.786443 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:30.786444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:30.786452 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:30.786512 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:30.786670 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:30.787225 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:30.788246 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:30.788544 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:30.788621 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:30.789342 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:30.789344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:30.789344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:30.789348 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:30.789364 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:30.789368 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:30.789433 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:30.789555 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 04:20:30.789565 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 04:20:30.789688 7212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:30.791244 7212 catalog_manager.cc:1347] Generated new cluster ID: 17ec772e43c547d1b5d76ff3b46d3c5c
I20260810 04:20:30.791437 7212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:30.797533 7212 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:30.798183 7212 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:30.800339 7212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77ec26727fd74a9d9aa52f04e3c25017: Generated new TSK 0
I20260810 04:20:30.800516 7212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:30.805647 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7115
W20260810 04:20:30.812321 7098 connection.cc:535] client connection to 127.15.216.126:36849 recv error: Network error: failed to read from TLS socket (remote: 127.15.216.126:36849): Connection reset by peer (error 104)
2026-08-10T04:20:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:30 dist-test-slave-g350 krb5kdc[7227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:30 dist-test-slave-g350 krb5kdc[7227](info): set up 2 sockets
Aug 10 04:20:30 dist-test-slave-g350 krb5kdc[7227](info): commencing operation
krb5kdc: starting...
W20260810 04:20:31.966639 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:32 dist-test-slave-g350 krb5kdc[7227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:32.101763 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45125
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:32.185941 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:32.186178 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:32.186218 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:32.187408 7241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:32.187481 7241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:32.187506 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:32.187538 7241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:32.187582 7241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:32.189131 7241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:32.189433 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:32.189750 7241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:32.194886 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:32.194912 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:32.194983 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:32.194912 7241 server_base.cc:1034] running on GCE node
I20260810 04:20:32.195324 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:32.195583 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:32.196753 7241 hybrid_clock.cc:648] HybridClock initialized: now 1786335632196736 us; error 31 us; skew 500 ppm
Aug 10 04:20:32 dist-test-slave-g350 krb5kdc[7227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335632, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:32.199549 7241 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:32.200057 7241 webserver.cc:460] Webserver started at http://127.15.216.126:42493/ using document root <none> and password file <none>
I20260810 04:20:32.200304 7241 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:32.200371 7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:32.200521 7241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:32.201722 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2"
format_stamp: "Formatted at 2026-08-10 04:20:32 on dist-test-slave-g350"
server_key: "0a4f5bc762997eeeec530079058733ff"
server_key_iv: "d0362caa036cdcc9f4c5c1d626d80850"
server_key_version: "encryptionkey@0"
I20260810 04:20:32.202116 7241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2"
format_stamp: "Formatted at 2026-08-10 04:20:32 on dist-test-slave-g350"
server_key: "0a4f5bc762997eeeec530079058733ff"
server_key_iv: "d0362caa036cdcc9f4c5c1d626d80850"
server_key_version: "encryptionkey@0"
I20260810 04:20:32.203526 7241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 04:20:32.204336 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:32.204563 7241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 04:20:32.204671 7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2"
format_stamp: "Formatted at 2026-08-10 04:20:32 on dist-test-slave-g350"
server_key: "0a4f5bc762997eeeec530079058733ff"
server_key_iv: "d0362caa036cdcc9f4c5c1d626d80850"
server_key_version: "encryptionkey@0"
I20260810 04:20:32.204758 7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:32.208597 7241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:32.211048 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:32.211274 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:32.217177 7241 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:45125
I20260810 04:20:32.217209 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:45125 every 8 connection(s)
I20260810 04:20:32.217821 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:32.219265 7311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:32.223357 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2: Bootstrap starting.
I20260810 04:20:32.224161 7311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:32.224519 7311 log.cc:824] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:32.225399 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2: No bootstrap required, opened a new log
I20260810 04:20:32.227922 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7241
I20260810 04:20:32.228061 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:32.228271 16225 external_mini_cluster.cc:1428] Setting key 206571ed48b354c4c6792a532fad19d5
I20260810 04:20:32.228221 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:20:32.228430 7311 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:32.228483 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c83d0f94c5437a8f45b3e74aaee9c2, State: Initialized, Role: FOLLOWER
I20260810 04:20:32.228648 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:20:32.228754 7311 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:32.228794 7311 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:32.228849 7311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:32.229717 7311 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:20:32.229840 7311 leader_election.cc:302] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c83d0f94c5437a8f45b3e74aaee9c2; no voters:
I20260810 04:20:32.230042 7311 leader_election.cc:288] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:32.230078 7316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:32.230244 7316 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [term 1 LEADER]: Becoming Leader. State: Replica: e0c83d0f94c5437a8f45b3e74aaee9c2, State: Running, Role: LEADER
I20260810 04:20:32.230342 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:32.230701 7316 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } }
I20260810 04:20:32.231040 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:32.228766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51059 (local address 127.15.216.126:45125)
0810 04:20:32.229030 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:32.229040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:32.229078 (+ 38us) server_negotiation.cc:407] Connection header received
0810 04:20:32.229534 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:32.229543 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:32.229591 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:32.229781 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:32.230518 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:32.230522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:32.230522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:32.230526 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:32.230563 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:32.230574 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:32.230644 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:32.230796 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 04:20:32.230839 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":134,"threads_started":1}
I20260810 04:20:32.231071 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } } }
I20260810 04:20:32.231141 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:32.231271 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c83d0f94c5437a8f45b3e74aaee9c2. Latest consensus state: current_term: 1 leader_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c83d0f94c5437a8f45b3e74aaee9c2" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45125 } } }
I20260810 04:20:32.231323 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:32.232239 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 04:20:32.232492 7332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:20:32.232533 7332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:20:32.232614 7325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:32.232824 7325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:32.234898 7325 catalog_manager.cc:1347] Generated new cluster ID: 72ee2293897948459ce8c42e9517b041
I20260810 04:20:32.234974 7325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:32.235924 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:32.233935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38377 (local address 127.15.216.126:45125)
0810 04:20:32.233971 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:32.233973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:32.234702 (+ 729us) server_negotiation.cc:407] Connection header received
0810 04:20:32.234820 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:32.234821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:32.234827 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:32.234894 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:32.235660 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:32.235661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:32.235661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:32.235665 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:32.235680 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:32.235684 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:32.235716 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:32.235826 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 04:20:32.235840 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:20:32.243824 7325 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:32.244015 7325 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:32.255486 7325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c83d0f94c5437a8f45b3e74aaee9c2: Generated new TSK 0
I20260810 04:20:32.255725 7325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:32.260773 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7241
2026-08-10T04:20:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:32 dist-test-slave-g350 krb5kdc[7343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:32 dist-test-slave-g350 krb5kdc[7343](info): set up 2 sockets
Aug 10 04:20:32 dist-test-slave-g350 krb5kdc[7343](info): commencing operation
krb5kdc: starting...
W20260810 04:20:33.406842 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:33 dist-test-slave-g350 krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:33.533099 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40399
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:37801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:33.616995 7356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:33.617252 7356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:33.617295 7356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:33.618606 7356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:33.618682 7356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:33.618711 7356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:33.618744 7356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:33.618772 7356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:33.620388 7356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:37801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:40399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:33.620680 7356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:33.620926 7356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:33.626163 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:33.626142 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:33.626174 7356 server_base.cc:1034] running on GCE node
W20260810 04:20:33.626142 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:33.626614 7356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:33.626860 7356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:33.628022 7356 hybrid_clock.cc:648] HybridClock initialized: now 1786335633628003 us; error 42 us; skew 500 ppm
Aug 10 04:20:33 dist-test-slave-g350 krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335633, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:33.630618 7356 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:33.631039 7356 webserver.cc:460] Webserver started at http://127.15.216.126:43843/ using document root <none> and password file <none>
I20260810 04:20:33.631356 7356 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:33.631433 7356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:33.631585 7356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:33.632629 7356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "e0c63771e7a645ecac5360c60c7172e6"
format_stamp: "Formatted at 2026-08-10 04:20:33 on dist-test-slave-g350"
server_key: "38a6c7cb7b29c542ba362b916b325126"
server_key_iv: "b5f2b9498f0a2588f5fcedb78114c226"
server_key_version: "encryptionkey@0"
I20260810 04:20:33.633002 7356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "e0c63771e7a645ecac5360c60c7172e6"
format_stamp: "Formatted at 2026-08-10 04:20:33 on dist-test-slave-g350"
server_key: "38a6c7cb7b29c542ba362b916b325126"
server_key_iv: "b5f2b9498f0a2588f5fcedb78114c226"
server_key_version: "encryptionkey@0"
I20260810 04:20:33.634473 7356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260810 04:20:33.635211 7376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:33.635413 7356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:33.635524 7356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "e0c63771e7a645ecac5360c60c7172e6"
format_stamp: "Formatted at 2026-08-10 04:20:33 on dist-test-slave-g350"
server_key: "38a6c7cb7b29c542ba362b916b325126"
server_key_iv: "b5f2b9498f0a2588f5fcedb78114c226"
server_key_version: "encryptionkey@0"
I20260810 04:20:33.635603 7356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:33.646343 7356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:33.648388 7356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:33.648535 7356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:33.652727 7356 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:40399
I20260810 04:20:33.652797 7430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:40399 every 8 connection(s)
I20260810 04:20:33.653144 7356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:33.654125 7431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:33.656432 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6: Bootstrap starting.
I20260810 04:20:33.657032 7431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:33.657330 7431 log.cc:824] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:33.658140 7431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6: No bootstrap required, opened a new log
I20260810 04:20:33.658664 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7356
I20260810 04:20:33.658818 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:33.659025 16225 external_mini_cluster.cc:1428] Setting key 128cede15103ef68901c01bb41187b0c
I20260810 04:20:33.660243 7431 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c63771e7a645ecac5360c60c7172e6" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40399 } }
I20260810 04:20:33.660408 7431 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:33.660446 7431 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c63771e7a645ecac5360c60c7172e6, State: Initialized, Role: FOLLOWER
I20260810 04:20:33.660602 7431 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c63771e7a645ecac5360c60c7172e6" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40399 } }
I20260810 04:20:33.660696 7431 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:33.660742 7431 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:33.660791 7431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:33.661705 7431 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c63771e7a645ecac5360c60c7172e6" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40399 } }
I20260810 04:20:33.661823 7431 leader_election.cc:302] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0c63771e7a645ecac5360c60c7172e6; no voters:
I20260810 04:20:33.662006 7431 leader_election.cc:288] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:33.662055 7436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:33.662261 7436 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [term 1 LEADER]: Becoming Leader. State: Replica: e0c63771e7a645ecac5360c60c7172e6, State: Running, Role: LEADER
I20260810 04:20:33.662283 7431 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:33.662356 7436 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c63771e7a645ecac5360c60c7172e6" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40399 } }
I20260810 04:20:33.662737 7436 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0c63771e7a645ecac5360c60c7172e6. Latest consensus state: current_term: 1 leader_uuid: "e0c63771e7a645ecac5360c60c7172e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c63771e7a645ecac5360c60c7172e6" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40399 } } }
I20260810 04:20:33.663002 7436 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:33.662928 7437 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0c63771e7a645ecac5360c60c7172e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0c63771e7a645ecac5360c60c7172e6" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 40399 } } }
I20260810 04:20:33.663468 7437 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:33.663586 7447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:33.664088 7447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:33.665855 7447 catalog_manager.cc:1347] Generated new cluster ID: f79b6b99103c47828feca8707defe029
I20260810 04:20:33.665910 7447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 04:20:33 dist-test-slave-g350 krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:33.670622 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:33.659588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51971 (local address 127.15.216.126:40399)
0810 04:20:33.659771 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:33.659784 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:33.659810 (+ 26us) server_negotiation.cc:407] Connection header received
0810 04:20:33.660253 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:33.660262 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:33.660329 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:33.660519 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:33.661418 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:33.662749 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:33.663981 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:33.664547 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:33.664663 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:33.667333 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:33.667337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:33.667338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:33.667349 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:33.669063 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:33.669454 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:33.669457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:33.669458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:33.669558 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:33.669821 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:33.669822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:33.669823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:33.670061 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:33.670265 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:33.670332 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 04:20:33.670472 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260810 04:20:33.671885 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:33.677414 7447 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:33.678107 7447 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:33.679862 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:33.673289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59351 (local address 127.15.216.126:40399)
0810 04:20:33.673344 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:33.673346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:33.673764 (+ 418us) server_negotiation.cc:407] Connection header received
0810 04:20:33.673838 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:33.673839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:33.673851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:33.673923 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:33.674461 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:33.674941 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:33.675650 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:33.675876 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:33.675959 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:33.677223 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:33.677224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:33.677224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:33.677231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:33.678675 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:33.679047 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:33.679048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:33.679048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:33.679108 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:33.679315 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:33.679316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:33.679316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:33.679427 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:33.679527 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:33.679659 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 04:20:33.679751 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 04:20:33.689332 7447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0c63771e7a645ecac5360c60c7172e6: Generated new TSK 0
I20260810 04:20:33.689546 7447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:33.692152 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7356
2026-08-10T04:20:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:33 dist-test-slave-g350 krb5kdc[7466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:33 dist-test-slave-g350 krb5kdc[7466](info): set up 2 sockets
Aug 10 04:20:33 dist-test-slave-g350 krb5kdc[7466](info): commencing operation
krb5kdc: starting...
W20260810 04:20:34.771550 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:34 dist-test-slave-g350 krb5kdc[7466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:34.901010 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44865
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:44865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:34.983311 7479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:34.983557 7479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:34.983595 7479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:34.984730 7479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:34.984803 7479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:34.984841 7479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:34.984872 7479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:34.984896 7479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:34.986404 7479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:44865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:34.986689 7479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:34.986928 7479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:34.992209 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:34.992348 7479 server_base.cc:1034] running on GCE node
W20260810 04:20:34.992332 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:34.992461 7486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:34.992780 7479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:34.993049 7479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:34.994228 7479 hybrid_clock.cc:648] HybridClock initialized: now 1786335634994211 us; error 33 us; skew 500 ppm
Aug 10 04:20:34 dist-test-slave-g350 krb5kdc[7466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335634, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:35.001864 7479 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:35.002389 7479 webserver.cc:460] Webserver started at http://127.15.216.126:33295/ using document root <none> and password file <none>
I20260810 04:20:35.002624 7479 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:35.002694 7479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:35.002856 7479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:35.004037 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "81385e348e204a7fb370ed2a74166790"
format_stamp: "Formatted at 2026-08-10 04:20:34 on dist-test-slave-g350"
server_key: "db7e77d2f697eda17ea394fa7a922a1a"
server_key_iv: "3e1220b36bb0d2f51f0573eaaf7b458b"
server_key_version: "encryptionkey@0"
I20260810 04:20:35.004423 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "81385e348e204a7fb370ed2a74166790"
format_stamp: "Formatted at 2026-08-10 04:20:34 on dist-test-slave-g350"
server_key: "db7e77d2f697eda17ea394fa7a922a1a"
server_key_iv: "3e1220b36bb0d2f51f0573eaaf7b458b"
server_key_version: "encryptionkey@0"
I20260810 04:20:35.005910 7479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 04:20:35.006856 7499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:35.007079 7479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:35.007207 7479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "81385e348e204a7fb370ed2a74166790"
format_stamp: "Formatted at 2026-08-10 04:20:34 on dist-test-slave-g350"
server_key: "db7e77d2f697eda17ea394fa7a922a1a"
server_key_iv: "3e1220b36bb0d2f51f0573eaaf7b458b"
server_key_version: "encryptionkey@0"
I20260810 04:20:35.007345 7479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:35.023458 7479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:35.025789 7479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:35.025971 7479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:35.030850 7479 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:44865
I20260810 04:20:35.030915 7557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:44865 every 8 connection(s)
I20260810 04:20:35.031363 7479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:35.032387 7558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:35.035650 7558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790: Bootstrap starting.
I20260810 04:20:35.036365 7558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:35.036715 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7479
I20260810 04:20:35.036751 7558 log.cc:824] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:35.036882 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:35.037112 16225 external_mini_cluster.cc:1428] Setting key f1545df8dcbdc78b5489bed050b80030
I20260810 04:20:35.037703 7558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790: No bootstrap required, opened a new log
I20260810 04:20:35.039584 7558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81385e348e204a7fb370ed2a74166790" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44865 } }
I20260810 04:20:35.039827 7558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:35.039901 7558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81385e348e204a7fb370ed2a74166790, State: Initialized, Role: FOLLOWER
I20260810 04:20:35.040036 7558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81385e348e204a7fb370ed2a74166790" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44865 } }
I20260810 04:20:35.040143 7558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:35.040184 7558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:35.040246 7558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:35.041050 7558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81385e348e204a7fb370ed2a74166790" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44865 } }
I20260810 04:20:35.041175 7558 leader_election.cc:302] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81385e348e204a7fb370ed2a74166790; no voters:
I20260810 04:20:35.041345 7558 leader_election.cc:288] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:35.041422 7564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:35.041582 7558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:35.041638 7564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [term 1 LEADER]: Becoming Leader. State: Replica: 81385e348e204a7fb370ed2a74166790, State: Running, Role: LEADER
I20260810 04:20:35.041883 7564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81385e348e204a7fb370ed2a74166790" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44865 } }
I20260810 04:20:35.042337 7565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81385e348e204a7fb370ed2a74166790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81385e348e204a7fb370ed2a74166790" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44865 } } }
I20260810 04:20:35.042423 7565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:35.042831 7566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81385e348e204a7fb370ed2a74166790. Latest consensus state: current_term: 1 leader_uuid: "81385e348e204a7fb370ed2a74166790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81385e348e204a7fb370ed2a74166790" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44865 } } }
I20260810 04:20:35.042904 7566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:35.043206 7573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:35.043784 7573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:35.045399 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:35.037722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48039 (local address 127.15.216.126:44865)
0810 04:20:35.037906 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:35.037914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:35.037942 (+ 28us) server_negotiation.cc:407] Connection header received
0810 04:20:35.039532 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:35.039541 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:35.039602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:35.039735 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:35.040042 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:35.041784 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:35.043068 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:35.043646 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:35.043781 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:35.044627 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:35.044631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:35.044631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:35.044906 (+ 275us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:35.044952 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:35.044968 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:35.045046 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:35.045126 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 04:20:35.045194 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260810 04:20:35.045629 7573 catalog_manager.cc:1347] Generated new cluster ID: 52573e5c531947bdb721d94732df0418
I20260810 04:20:35.045718 7573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:35.046536 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:35.052858 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:35.048051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44837 (local address 127.15.216.126:44865)
0810 04:20:35.048108 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:35.048110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:35.049871 (+ 1761us) server_negotiation.cc:407] Connection header received
0810 04:20:35.049946 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:35.049947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:35.049960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:35.050017 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:35.050184 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:35.050679 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:35.051535 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:35.051755 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:35.051849 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:35.052602 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:35.052604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:35.052604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:35.052609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:35.052624 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:35.052629 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:35.052682 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:35.052745 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 04:20:35.052769 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 04:20:35.058835 7573 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:35.059593 7573 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:35.063732 7573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81385e348e204a7fb370ed2a74166790: Generated new TSK 0
I20260810 04:20:35.063920 7573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:35.067550 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7479
2026-08-10T04:20:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:35 dist-test-slave-g350 krb5kdc[7592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:35 dist-test-slave-g350 krb5kdc[7592](info): set up 2 sockets
Aug 10 04:20:35 dist-test-slave-g350 krb5kdc[7592](info): commencing operation
krb5kdc: starting...
W20260810 04:20:36.143383 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:36 dist-test-slave-g350 krb5kdc[7592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:36.276640 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44077
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:44077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:36.361974 7605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:36.362224 7605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:36.362277 7605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:36.363438 7605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:36.363507 7605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:36.363533 7605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:36.363561 7605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:36.363592 7605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:36.365064 7605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:44077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:44077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:36.365350 7605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:36.365578 7605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:20:36.370755 7605 server_base.cc:1034] running on GCE node
W20260810 04:20:36.370682 7611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:36.370714 7612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:36.370705 7614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:36.371213 7605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:36.371515 7605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:36.372656 7605 hybrid_clock.cc:648] HybridClock initialized: now 1786335636372624 us; error 47 us; skew 500 ppm
Aug 10 04:20:36 dist-test-slave-g350 krb5kdc[7592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335636, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:36.375245 7605 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:36.375679 7605 webserver.cc:460] Webserver started at http://127.15.216.126:33119/ using document root <none> and password file <none>
I20260810 04:20:36.375941 7605 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:36.376012 7605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:36.376160 7605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:36.377219 7605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "8c944891aa724ff3a6a2efdd9ecbd178"
format_stamp: "Formatted at 2026-08-10 04:20:36 on dist-test-slave-g350"
server_key: "259961987df3a9114001394a85e4876f"
server_key_iv: "a81e37de96287bf85461d4d64c2285d2"
server_key_version: "encryptionkey@0"
I20260810 04:20:36.377630 7605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "8c944891aa724ff3a6a2efdd9ecbd178"
format_stamp: "Formatted at 2026-08-10 04:20:36 on dist-test-slave-g350"
server_key: "259961987df3a9114001394a85e4876f"
server_key_iv: "a81e37de96287bf85461d4d64c2285d2"
server_key_version: "encryptionkey@0"
I20260810 04:20:36.378996 7605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:36.379932 7621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:36.380144 7605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:36.380254 7605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "8c944891aa724ff3a6a2efdd9ecbd178"
format_stamp: "Formatted at 2026-08-10 04:20:36 on dist-test-slave-g350"
server_key: "259961987df3a9114001394a85e4876f"
server_key_iv: "a81e37de96287bf85461d4d64c2285d2"
server_key_version: "encryptionkey@0"
I20260810 04:20:36.380333 7605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:36.383747 7605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:36.386071 7605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:36.386233 7605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:36.390390 7605 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:44077
I20260810 04:20:36.390420 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:44077 every 8 connection(s)
I20260810 04:20:36.390770 7605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:36.392040 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:36.393337 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7605
I20260810 04:20:36.393519 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:36.393792 16225 external_mini_cluster.cc:1428] Setting key 0fb34bb257d9833b6a2b1360afcead45
I20260810 04:20:36.395057 7680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178: Bootstrap starting.
I20260810 04:20:36.395816 7680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:36.396206 7680 log.cc:824] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:36.396657 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:36.394428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42567 (local address 127.15.216.126:44077)
0810 04:20:36.394643 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:36.394652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:36.394701 (+ 49us) server_negotiation.cc:407] Connection header received
0810 04:20:36.395259 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:36.395268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:36.395301 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:36.395471 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:36.396134 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:36.396137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:36.396137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:36.396141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:36.396175 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:36.396185 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:36.396255 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:36.396357 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 04:20:36.396438 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260810 04:20:36.397143 7680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178: No bootstrap required, opened a new log
I20260810 04:20:36.397802 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:36.399600 7680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44077 } }
I20260810 04:20:36.399763 7680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 04:20:36.399814 7680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c944891aa724ff3a6a2efdd9ecbd178, State: Initialized, Role: FOLLOWER
I20260810 04:20:36.399940 7680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44077 } }
I20260810 04:20:36.400067 7680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 04:20:36.400105 7680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 04:20:36.400161 7680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:36.400511 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:36.399199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41777 (local address 127.15.216.126:44077)
0810 04:20:36.399233 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:36.399235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:36.399490 (+ 255us) server_negotiation.cc:407] Connection header received
0810 04:20:36.399517 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:36.399518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:36.399522 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:36.399582 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:36.400283 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:36.400285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:36.400285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 04:20:36.400290 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:36.400304 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 04:20:36.400309 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:36.400361 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:36.400456 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 04:20:36.400469 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:20:36.400947 7680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44077 } }
I20260810 04:20:36.401093 7680 leader_election.cc:302] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c944891aa724ff3a6a2efdd9ecbd178; no voters:
I20260810 04:20:36.401274 7680 leader_election.cc:288] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 04:20:36.401331 7690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:36.401481 7690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [term 1 LEADER]: Becoming Leader. State: Replica: 8c944891aa724ff3a6a2efdd9ecbd178, State: Running, Role: LEADER
I20260810 04:20:36.401523 7680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:36.401640 7690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44077 } }
I20260810 04:20:36.402056 7692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c944891aa724ff3a6a2efdd9ecbd178. Latest consensus state: current_term: 1 leader_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44077 } } }
I20260810 04:20:36.402148 7692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:36.402359 7699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:36.402364 7691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c944891aa724ff3a6a2efdd9ecbd178" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 44077 } } }
I20260810 04:20:36.402714 7691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:36.403008 7699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:36.404527 7699 catalog_manager.cc:1347] Generated new cluster ID: 91e7f3a41ecf42e89cced2c019257283
I20260810 04:20:36.404577 7699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:36.412628 7699 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:36.412797 7699 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:36.420600 7699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c944891aa724ff3a6a2efdd9ecbd178: Generated new TSK 0
I20260810 04:20:36.420782 7699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:36.424801 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7605
2026-08-10T04:20:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1356 ms)
[----------] 18 tests from AuthTokenIssuingTest (25045 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:36 dist-test-slave-g350 krb5kdc[7712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:36 dist-test-slave-g350 krb5kdc[7712](info): set up 2 sockets
Aug 10 04:20:36 dist-test-slave-g350 krb5kdc[7712](info): commencing operation
krb5kdc: starting...
W20260810 04:20:37.506762 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:37 dist-test-slave-g350 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:37.636511 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45493
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45493,127.15.216.125:35919,127.15.216.124:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:37.720753 7727 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:20:37.720968 7727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:37.720988 7727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:37.721006 7727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:37.722280 7727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:37.722323 7727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:37.722337 7727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:37.722350 7727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:37.722398 7727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:37.723861 7727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45493,127.15.216.125:35919,127.15.216.124:36307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:37.724113 7727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:37.724344 7727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:37.729588 7733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:37.729662 7732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:37.729650 7735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:37.729693 7727 server_base.cc:1034] running on GCE node
I20260810 04:20:37.730036 7727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:37.730319 7727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:37.731485 7727 hybrid_clock.cc:648] HybridClock initialized: now 1786335637731459 us; error 38 us; skew 500 ppm
Aug 10 04:20:37 dist-test-slave-g350 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:37.734109 7727 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:37.734607 7727 webserver.cc:460] Webserver started at http://127.15.216.126:45531/ using document root <none> and password file <none>
I20260810 04:20:37.734831 7727 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:37.734876 7727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:37.735002 7727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:37.736044 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "d45c58beb16747f18e9a33dc1d59b157"
format_stamp: "Formatted at 2026-08-10 04:20:37 on dist-test-slave-g350"
server_key: "c896de7539473a819d0e453ea6a05157"
server_key_iv: "c5d55499c0966313bd78ec12c8244b8a"
server_key_version: "encryptionkey@0"
I20260810 04:20:37.736416 7727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "d45c58beb16747f18e9a33dc1d59b157"
format_stamp: "Formatted at 2026-08-10 04:20:37 on dist-test-slave-g350"
server_key: "c896de7539473a819d0e453ea6a05157"
server_key_iv: "c5d55499c0966313bd78ec12c8244b8a"
server_key_version: "encryptionkey@0"
I20260810 04:20:37.737870 7727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 04:20:37.738662 7742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:37.738849 7727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:37.738960 7727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "d45c58beb16747f18e9a33dc1d59b157"
format_stamp: "Formatted at 2026-08-10 04:20:37 on dist-test-slave-g350"
server_key: "c896de7539473a819d0e453ea6a05157"
server_key_iv: "c5d55499c0966313bd78ec12c8244b8a"
server_key_version: "encryptionkey@0"
I20260810 04:20:37.739035 7727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:37.765251 7727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:37.767439 7727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:37.767766 7727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:37.772245 7727 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:45493
I20260810 04:20:37.772279 7796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:45493 every 8 connection(s)
I20260810 04:20:37.772742 7727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:37.773696 7798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:37.775300 7798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:37 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:37.782624 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7727
I20260810 04:20:37.782719 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:37.782917 16225 external_mini_cluster.cc:1428] Setting key e2bcf45f136d10abb7246f148c8a7b7d
I20260810 04:20:37.785260 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.776025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45493 (local address 127.0.0.1:33204)
0810 04:20:37.776214 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.776253 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:37.776767 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:37.777033 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:37.777036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:37.777288 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:37.777445 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:37.777448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.778522 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.778523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:37.779202 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:37.779204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.779712 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.779712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:37.779783 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:37.780251 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:37.780261 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:37.781697 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:37.783719 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:37.783727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:37.783729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:37.784023 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:37.784259 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:37.784259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:37.784260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:37.784376 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:37.784710 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:37.784717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:37.784804 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 04:20:37.784915 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 04:20:37.785092 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260810 04:20:37.785378 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.776083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.15.216.126:45493)
0810 04:20:37.776250 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:37.776258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:37.776336 (+ 78us) server_negotiation.cc:407] Connection header received
0810 04:20:37.776896 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:37.776898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:37.776915 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:37.777006 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:37.777532 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.778417 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.779290 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.779632 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.779715 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:37.781799 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:37.781807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:37.781807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:37.781813 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:37.783635 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:37.784110 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:37.784112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:37.784112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:37.784191 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:37.784444 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:37.784445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:37.784445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:37.784644 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:37.784706 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:37.784981 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 04:20:37.785207 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260810 04:20:37.786271 7798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:20:37.786669 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.786542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57272)
0810 04:20:37.786593 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.786638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:20:37.786813 7743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
I20260810 04:20:37.787249 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.787157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57284)
0810 04:20:37.787203 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.787227 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:20:37.787413 7798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:35919: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.15.216.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab.
Entry for principal kudu/127.15.216.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab.
I20260810 04:20:37.812131 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:35919
--webserver_interface=127.15.216.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab
--principal=kudu/127.15.216.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45493,127.15.216.125:35919,127.15.216.124:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:20:37.836685 7798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } attempt: 1
I20260810 04:20:37.837265 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.837106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57290)
0810 04:20:37.837180 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.837229 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 04:20:37.837808 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.837705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57302)
0810 04:20:37.837755 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.837787 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:20:37.837915 7798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:35919: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
W20260810 04:20:37.896561 7805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:20:37.896785 7805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:37.896802 7805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:37.896824 7805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:37.898389 7805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:37.898442 7805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:37.898456 7805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:37.898469 7805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:37.898509 7805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:37.900055 7805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45493,127.15.216.125:35919,127.15.216.124:36307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab
--principal=kudu/127.15.216.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:35919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.216.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:37.900326 7805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:37.900563 7805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:20:37.905877 7805 server_base.cc:1034] running on GCE node
W20260810 04:20:37.905893 7812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:37.905936 7816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:37.906054 7798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } attempt: 2
W20260810 04:20:37.906167 7813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:37.906524 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.906336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57308)
0810 04:20:37.906422 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.906483 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 04:20:37.906679 7805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:37.906869 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.906781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57322)
0810 04:20:37.906814 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.906844 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 04:20:37.906981 7798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:35919: Network error: Client connection negotiation failed: client connection to 127.15.216.125:35919: connect: Connection refused (error 111)
I20260810 04:20:37.907021 7805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:37.908172 7805 hybrid_clock.cc:648] HybridClock initialized: now 1786335637908162 us; error 28 us; skew 500 ppm
Aug 10 04:20:37 dist-test-slave-g350 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:37.910856 7805 init.cc:377] Logged in from keytab as kudu/127.15.216.125@KRBTEST.COM (short username kudu)
I20260810 04:20:37.911300 7805 webserver.cc:460] Webserver started at http://127.15.216.125:39901/ using document root <none> and password file <none>
I20260810 04:20:37.911509 7805 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:37.911550 7805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:37.911638 7805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:37.912634 7805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data/instance:
uuid: "0a1e93ea68d5400693181df052ced552"
format_stamp: "Formatted at 2026-08-10 04:20:37 on dist-test-slave-g350"
server_key: "0faca808679d4e2077b6c83f6e5a12d7"
server_key_iv: "9e42f5952aab87f1d70d4ecbee9d7c0a"
server_key_version: "encryptionkey@0"
I20260810 04:20:37.912936 7805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/wal/instance:
uuid: "0a1e93ea68d5400693181df052ced552"
format_stamp: "Formatted at 2026-08-10 04:20:37 on dist-test-slave-g350"
server_key: "0faca808679d4e2077b6c83f6e5a12d7"
server_key_iv: "9e42f5952aab87f1d70d4ecbee9d7c0a"
server_key_version: "encryptionkey@0"
I20260810 04:20:37.914374 7805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 04:20:37.915206 7823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:37.915514 7805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:37.915629 7805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/wal
uuid: "0a1e93ea68d5400693181df052ced552"
format_stamp: "Formatted at 2026-08-10 04:20:37 on dist-test-slave-g350"
server_key: "0faca808679d4e2077b6c83f6e5a12d7"
server_key_iv: "9e42f5952aab87f1d70d4ecbee9d7c0a"
server_key_version: "encryptionkey@0"
I20260810 04:20:37.915709 7805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:37.923902 7805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:37.926306 7805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:37.926476 7805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:37.930724 7805 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.125:35919
I20260810 04:20:37.930770 7880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.125:35919 every 8 connection(s)
I20260810 04:20:37.931149 7805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
I20260810 04:20:37.932018 7881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:37.933640 7881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:20:37.937811 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7805
I20260810 04:20:37.937896 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-1/wal/instance
I20260810 04:20:37.938073 16225 external_mini_cluster.cc:1428] Setting key 258682224db7640a5d9ce215447038fd
Aug 10 04:20:37 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:37.944386 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.934386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.15.216.126:45493)
0810 04:20:37.934457 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:37.934459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:37.934710 (+ 251us) server_negotiation.cc:407] Connection header received
0810 04:20:37.935177 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:37.935178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:37.935191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:37.935245 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:37.935876 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.936376 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.937338 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.937546 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.937652 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:37.940433 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:37.940434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:37.940435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:37.940443 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:37.942451 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:37.943096 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:37.943097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:37.943098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:37.943179 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:37.943465 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:37.943466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:37.943466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:37.943673 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:37.943737 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:37.944195 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 04:20:37.944286 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 04:20:37.944988 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.934425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45493 (local address 127.0.0.1:33214)
0810 04:20:37.934593 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.934627 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:37.935092 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:37.935353 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:37.935356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:37.935605 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:37.935795 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:37.935798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.936466 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.936467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:37.937259 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:37.937261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.937687 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.937688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:37.937791 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:37.938397 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:37.938408 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:37.940311 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:37.942553 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:37.942590 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:37.942593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:37.943002 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:37.943245 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:37.943246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:37.943247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:37.943383 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:37.943740 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:37.943773 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:37.944026 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 04:20:37.944491 (+ 465us) client_negotiation.cc:240] Negotiation successful
0810 04:20:37.944677 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260810 04:20:37.945830 7881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.216.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.124@KRBTEST.COM" created.
Aug 10 04:20:37 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:20:37.956346 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.946124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57328)
0810 04:20:37.946417 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.946430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:37.946487 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:37.946700 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:37.946701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:37.946874 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:37.946940 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:37.946942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.947794 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.947795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:37.948644 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:37.948647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.949293 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.949294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:37.949384 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:37.949829 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:37.949833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:37.953206 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:37.955077 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:37.955078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:37.955079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:37.955318 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:37.955540 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:37.955541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:37.955541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:37.955624 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:37.956058 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:37.956059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:37.956099 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 04:20:37.956170 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:20:37.956275 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260810 04:20:37.956513 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.946200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.15.216.125:35919)
0810 04:20:37.946412 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:37.946416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:37.946494 (+ 78us) server_negotiation.cc:407] Connection header received
0810 04:20:37.946567 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:37.946571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:37.946585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:37.946701 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:37.947014 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.947693 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.948778 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.949209 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.949274 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:37.953311 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:37.953319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:37.953320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:37.953326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:37.954987 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:37.955414 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:37.955417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:37.955417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:37.955471 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:37.955712 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:37.955713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:37.955713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:37.955938 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:37.956033 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:37.956227 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 04:20:37.956352 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260810 04:20:37.957129 7881 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:20:37.957554 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.957434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60142)
0810 04:20:37.957485 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.957535 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 04:20:37.957715 7826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
I20260810 04:20:37.958151 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.958058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60158)
0810 04:20:37.958102 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.958126 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:20:37.958253 7881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:36307: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab.
Entry for principal kudu/127.15.216.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab.
I20260810 04:20:37.969012 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:36307
--webserver_interface=127.15.216.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab
--principal=kudu/127.15.216.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:45493,127.15.216.125:35919,127.15.216.124:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:20:37.998158 7798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } attempt: 3
Aug 10 04:20:37 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:20:38.005429 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.998560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57332)
0810 04:20:37.998635 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:37.998646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:37.998691 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:37.998910 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:37.998912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:37.999079 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:37.999167 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:37.999168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.999903 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:37.999905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.000442 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.000444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.000839 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.000840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.000923 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.001402 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:38.001415 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:38.002715 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:38.004341 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.004342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.004343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.004512 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.004737 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.004738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.004739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.004812 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.005077 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:38.005078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:38.005124 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:20:38.005222 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 04:20:38.005325 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 04:20:38.005429 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:37.998665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.15.216.125:35919)
0810 04:20:37.998754 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:37.998756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:37.998765 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:37.998801 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:37.998802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:37.998812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:37.998877 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:37.999312 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:37.999809 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.000532 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.000761 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.000842 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.002805 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.002807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.002808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.002816 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.004252 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.004600 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.004602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.004602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.004662 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.004883 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.004884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.004884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.004996 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.005066 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.005250 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.005327 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 04:20:38.005909 7798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:20:38.006274 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.006162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60164)
0810 04:20:38.006219 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.006247 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 04:20:38.006628 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.006538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60166)
0810 04:20:38.006584 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.006608 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:20:38.006726 7798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:36307: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
I20260810 04:20:38.007463 7881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } attempt: 1
I20260810 04:20:38.007812 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.007707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60178)
0810 04:20:38.007757 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.007784 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 04:20:38.008177 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.008045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60184)
0810 04:20:38.008106 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.008146 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 04:20:38.008322 7881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:36307: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
I20260810 04:20:38.037866 7798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } attempt: 1
I20260810 04:20:38.038409 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.038230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60198)
0810 04:20:38.038307 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.038366 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 04:20:38.038849 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.038742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60210)
0810 04:20:38.038793 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.038825 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:20:38.039000 7798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:36307: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
W20260810 04:20:38.055168 7889 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:20:38.055373 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:38.055390 7889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:38.055410 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:38.056617 7889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:38.056659 7889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:38.056674 7889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:38.056686 7889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:38.056699 7889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:38.058326 7889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:45493,127.15.216.125:35919,127.15.216.124:36307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab
--principal=kudu/127.15.216.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.216.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:38.058614 7889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:38.058866 7889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:38.063836 7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:38.063843 7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:38.063951 7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:38.064198 7889 server_base.cc:1034] running on GCE node
I20260810 04:20:38.064468 7889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:38.064770 7889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:38.065963 7889 hybrid_clock.cc:648] HybridClock initialized: now 1786335638065944 us; error 41 us; skew 500 ppm
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:38.068637 7889 init.cc:377] Logged in from keytab as kudu/127.15.216.124@KRBTEST.COM (short username kudu)
I20260810 04:20:38.069144 7889 webserver.cc:460] Webserver started at http://127.15.216.124:33811/ using document root <none> and password file <none>
I20260810 04:20:38.069403 7889 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:38.069475 7889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:38.069679 7889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:38.070807 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data/instance:
uuid: "ee19159eda4447eca485131e9032ee93"
format_stamp: "Formatted at 2026-08-10 04:20:38 on dist-test-slave-g350"
server_key: "37deee79693e6912097cb6a7a574c894"
server_key_iv: "d6732b8ec044ec8ee46d78dabe7d354c"
server_key_version: "encryptionkey@0"
I20260810 04:20:38.071200 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/wal/instance:
uuid: "ee19159eda4447eca485131e9032ee93"
format_stamp: "Formatted at 2026-08-10 04:20:38 on dist-test-slave-g350"
server_key: "37deee79693e6912097cb6a7a574c894"
server_key_iv: "d6732b8ec044ec8ee46d78dabe7d354c"
server_key_version: "encryptionkey@0"
I20260810 04:20:38.072661 7889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 04:20:38.073700 7904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:38.073897 7889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:38.074024 7889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/wal
uuid: "ee19159eda4447eca485131e9032ee93"
format_stamp: "Formatted at 2026-08-10 04:20:38 on dist-test-slave-g350"
server_key: "37deee79693e6912097cb6a7a574c894"
server_key_iv: "d6732b8ec044ec8ee46d78dabe7d354c"
server_key_version: "encryptionkey@0"
I20260810 04:20:38.074122 7889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:38.076489 7881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } attempt: 2
I20260810 04:20:38.076977 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.076818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60220)
0810 04:20:38.076892 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.076938 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 04:20:38.077349 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.077235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60230)
0810 04:20:38.077287 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.077322 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 04:20:38.077464 7881 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:36307: Network error: Client connection negotiation failed: client connection to 127.15.216.124:36307: connect: Connection refused (error 111)
I20260810 04:20:38.089097 7889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:38.091249 7889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:38.091418 7889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:38.095847 7889 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.124:36307
I20260810 04:20:38.095923 7963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.124:36307 every 8 connection(s)
I20260810 04:20:38.096253 7889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
I20260810 04:20:38.097146 7964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:38.098733 7964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:38.104859 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 7889
I20260810 04:20:38.104947 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786335362448781-16225-0/minicluster-data/master-2/wal/instance
I20260810 04:20:38.105139 16225 external_mini_cluster.cc:1428] Setting key 1df4c4534314433823569c8d8f5ee2be
I20260810 04:20:38.107782 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.099457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.15.216.126:45493)
0810 04:20:38.099510 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.099512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.099736 (+ 224us) server_negotiation.cc:407] Connection header received
0810 04:20:38.100192 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.100193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.100202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.100262 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.100834 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.101298 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.102319 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.102514 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.102573 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.104729 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.104731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.104731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.104738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.106320 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.106832 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.106833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.106834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.106904 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.107148 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.107149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.107150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.107288 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.107361 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.107595 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.107683 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:20:38.108117 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.099438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:45493 (local address 127.0.0.1:33230)
0810 04:20:38.099612 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.099660 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:38.100111 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:38.100312 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:38.100315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:38.100560 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:38.100746 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.100750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.101388 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.101389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.102230 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.102233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.102591 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.102591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.102688 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.103124 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:38.103139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:38.104623 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:38.106397 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.106406 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.106408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.106717 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.106973 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.106974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.106974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.107078 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.107357 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:38.107362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:38.107470 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 04:20:38.107551 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 04:20:38.107889 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":155,"thread_start_us":87,"threads_started":1}
I20260810 04:20:38.108695 7964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:20:38.114125 7798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } attempt: 2
I20260810 04:20:38.114717 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.105574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.15.216.124:36307)
0810 04:20:38.105762 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.105767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.105922 (+ 155us) server_negotiation.cc:407] Connection header received
0810 04:20:38.105992 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.105994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.106015 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.106089 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.106847 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.107926 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.108907 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.109272 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.109354 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.111671 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.111679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.111680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.111686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.113364 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.113791 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.113795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.113795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.113857 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.114070 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.114071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.114072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.114256 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.114450 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.114506 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.114586 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":123,"threads_started":1}
I20260810 04:20:38.115435 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.108976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:35919 (local address 127.0.0.1:57344)
0810 04:20:38.109023 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.109034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:38.109094 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:38.109225 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:38.109226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:38.109358 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:38.109421 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.109422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.110101 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.110102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.110583 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.110584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.110972 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.110973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.111052 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.111446 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:38.111451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:38.112550 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:38.114287 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.114289 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.114289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.114513 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.114736 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.114737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.114737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.114802 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.115123 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:38.115124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:38.115168 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 04:20:38.115239 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 04:20:38.115328 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 04:20:38.115794 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.109041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.15.216.125:35919)
0810 04:20:38.109114 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.109117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.109129 (+ 12us) server_negotiation.cc:407] Connection header received
0810 04:20:38.109154 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.109155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.109165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.109221 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.109500 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.110022 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.110668 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.110897 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.110974 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.112630 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.112631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.112632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.112639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.114132 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.114597 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.114599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.114599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.114662 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.114886 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.114887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.114888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.115027 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.115118 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.115623 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.115704 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 04:20:38.116117 7964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335638, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
I20260810 04:20:38.123282 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.114393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60246)
0810 04:20:38.114483 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.114494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:38.114532 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:38.114899 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:38.114900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:38.115063 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:38.115107 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.115109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.116054 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.116055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.117329 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.117332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.117818 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.117819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.117889 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.118281 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:38.118286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:38.119744 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:38.121990 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.121991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.121992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.122231 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.122467 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.122469 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.122469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.122540 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.122882 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:38.122883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:38.122941 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 04:20:38.122999 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 04:20:38.123109 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 04:20:38.123504 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.114604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.15.216.124:36307)
0810 04:20:38.114756 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.114758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.114765 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:20:38.114782 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.114782 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.114792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.115181 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.115241 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.115965 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.117407 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.117742 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.117819 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.119830 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.119831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.119831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.119838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.121909 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.122315 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.122316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.122317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.122373 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.122624 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.122625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.122625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.122768 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.123231 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.123274 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.123348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260810 04:20:38.124123 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.115653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.15.216.126:45493)
0810 04:20:38.115690 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.115691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.115699 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:38.115754 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.115754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.115764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.115825 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.116413 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.117081 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.118110 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.118402 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.118459 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.121115 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.121117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.121117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.121124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.122791 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.123140 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.123141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.123142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.123215 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.123450 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.123451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.123451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.123567 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.123735 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.123942 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.124024 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:20:38.125519 7798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157: Bootstrap starting.
I20260810 04:20:38.125650 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.116341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:60258)
0810 04:20:38.117425 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.117436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:38.117480 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:38.117649 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:38.117650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:38.117789 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:38.117827 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.117829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.118655 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.118656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.119414 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.119417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.119876 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.119877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.119927 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.120406 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:38.120411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:38.121916 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:38.124439 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.124440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.124440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.124676 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.124883 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.124884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.124884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.124953 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.125293 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:38.125294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:38.125350 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 04:20:38.125428 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 04:20:38.125524 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058}
I20260810 04:20:38.125972 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.116422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.15.216.124:36307)
0810 04:20:38.116451 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.116453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.117490 (+ 1037us) server_negotiation.cc:407] Connection header received
0810 04:20:38.117529 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.117530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.117540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.117643 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.117900 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.118577 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.119506 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.119820 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.119877 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.121997 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.121998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.121998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.122003 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.124363 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.124752 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.124753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.124753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.124806 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.125028 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.125029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.125029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.125204 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.125287 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.125474 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.125883 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":14}
I20260810 04:20:38.126430 7798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:38.126724 7798 log.cc:824] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:38.127353 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: Bootstrap starting.
I20260810 04:20:38.127547 7798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157: No bootstrap required, opened a new log
I20260810 04:20:38.128136 7964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:38.128491 7964 log.cc:824] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:38.129335 7964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: No bootstrap required, opened a new log
I20260810 04:20:38.129645 7798 raft_consensus.cc:348] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
I20260810 04:20:38.129798 7798 raft_consensus.cc:729] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45c58beb16747f18e9a33dc1d59b157, State: Initialized, Role: FOLLOWER
I20260810 04:20:38.129880 7798 consensus_queue.cc:260] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:20:38.130632 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.130719 7798 sys_catalog.cc:564] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:38.130743 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:38.131222 7964 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
I20260810 04:20:38.131390 7964 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee19159eda4447eca485131e9032ee93, State: Initialized, Role: FOLLOWER
I20260810 04:20:38.131494 7964 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
I20260810 04:20:38.131794 7964 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:38.131759 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.131845 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 04:20:38.132898 7988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:20:38.132956 7988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:20:38.134471 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.124772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.15.216.125:35919)
0810 04:20:38.124813 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.124815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.124822 (+ 7us) server_negotiation.cc:407] Connection header received
0810 04:20:38.124845 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.124845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.124856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.124905 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.125587 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.126311 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.127228 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.127525 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.127606 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.130542 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.130544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.130544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.130550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.132784 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.133286 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.133287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.133288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.133383 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.133616 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.133617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.133618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.133753 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.133912 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.134301 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.134379 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:20:38.133895 7727 master_runner.cc:186] Error getting master registration for 127.15.216.125:35919: OK, instance_id { permanent_uuid: "0a1e93ea68d5400693181df052ced552" instance_seqno: 1786335637929457 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:20:38.137022 7727 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:20:38.137643 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 04:20:38.137961 7999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:20:38.138020 7999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:20:38.135208 7889 master_runner.cc:186] Error getting master registration for 127.15.216.125:35919: OK, instance_id { permanent_uuid: "0a1e93ea68d5400693181df052ced552" instance_seqno: 1786335637929457 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:20:38.138142 7889 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:20:38.138675 7769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d45c58beb16747f18e9a33dc1d59b157"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33240
I20260810 04:20:38.138799 7769 raft_consensus.cc:482] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 04:20:38.138857 7769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:38.139606 7769 raft_consensus.cc:504] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
I20260810 04:20:38.139847 7769 leader_election.cc:288] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [CANDIDATE]: Term 1 election: Requested vote from peers 0a1e93ea68d5400693181df052ced552 (127.15.216.125:35919), ee19159eda4447eca485131e9032ee93 (127.15.216.124:36307)
I20260810 04:20:38.140296 7937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d45c58beb16747f18e9a33dc1d59b157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ee19159eda4447eca485131e9032ee93"
I20260810 04:20:38.140457 7937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:38.141515 7937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45c58beb16747f18e9a33dc1d59b157 in term 1.
I20260810 04:20:38.141743 7744 leader_election.cc:302] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d45c58beb16747f18e9a33dc1d59b157, ee19159eda4447eca485131e9032ee93; no voters:
I20260810 04:20:38.141889 7972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:38.142048 7972 raft_consensus.cc:686] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [term 1 LEADER]: Becoming Leader. State: Replica: d45c58beb16747f18e9a33dc1d59b157, State: Running, Role: LEADER
I20260810 04:20:38.142151 7972 consensus_queue.cc:237] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
I20260810 04:20:38.142606 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45c58beb16747f18e9a33dc1d59b157. Latest consensus state: current_term: 1 leader_uuid: "d45c58beb16747f18e9a33dc1d59b157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.142725 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:38.143682 8009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:38.144114 8009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:38.151594 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.141122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.15.216.126:45493)
0810 04:20:38.141165 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.141167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.143336 (+ 2169us) server_negotiation.cc:407] Connection header received
0810 04:20:38.143460 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.143462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.143474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.143507 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.144415 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.145109 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.146309 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.146652 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.146701 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.148355 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.148357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.148357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.148366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.150145 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.150599 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.150601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.150601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.150664 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.150886 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.150887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.150888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.151027 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.151136 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.151394 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.151478 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:20:38.156245 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.141469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.15.216.124:36307)
0810 04:20:38.141813 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.142141 (+ 328us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.145359 (+ 3218us) server_negotiation.cc:407] Connection header received
0810 04:20:38.145698 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.145699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.145711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.145741 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.147031 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.147742 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.149386 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.149745 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.149792 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.152466 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.152467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.152468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.152476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.154377 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.154969 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.154981 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.154982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.155043 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.155346 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.155347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.155348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.155501 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.155601 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.156016 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.156107 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260810 04:20:38.159274 7937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [term 1 FOLLOWER]: Refusing update from remote peer d45c58beb16747f18e9a33dc1d59b157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:38.159792 7972 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:38.159888 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.143040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.15.216.125:35919)
0810 04:20:38.143083 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.143085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.152396 (+ 9311us) server_negotiation.cc:407] Connection header received
0810 04:20:38.152502 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.152504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.152518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.152550 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.153150 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.153930 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.154950 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.155266 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.155311 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.156752 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.156754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.156754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.156763 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.158507 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.158919 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.158921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.158921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.158985 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.159211 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.159212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.159212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.159399 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.159479 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.159712 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.159803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:20:38.161885 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45c58beb16747f18e9a33dc1d59b157. Latest consensus state: current_term: 1 leader_uuid: "d45c58beb16747f18e9a33dc1d59b157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.161993 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:38.168563 7881 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } attempt: 3
Aug 10 04:20:38 dist-test-slave-g350 krb5kdc[7712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335637, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
I20260810 04:20:38.175702 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45c58beb16747f18e9a33dc1d59b157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.175814 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:38.176718 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45c58beb16747f18e9a33dc1d59b157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.176800 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:38.177670 8011 mvcc.cc:205] Tried to move back new op lower bound from 7316830773896155136 to 7316830773831008256. Current Snapshot: MvccSnapshot[applied={T|T < 7316830773896155136}]
I20260810 04:20:38.177824 8012 mvcc.cc:205] Tried to move back new op lower bound from 7316830773896155136 to 7316830773831008256. Current Snapshot: MvccSnapshot[applied={T|T < 7316830773896155136 or (T in {7316830773896155136})}]
I20260810 04:20:38.178056 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.168910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.15.216.124:36307)
0810 04:20:38.168948 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.168950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.168958 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:38.168981 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.168982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.168991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.169038 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.169428 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.170201 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.171150 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.171451 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.171512 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.173788 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:38.173790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:38.173790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:38.173814 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:38.175753 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.176277 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.176279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.176279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.176366 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:38.177333 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:38.177334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:38.177334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:38.177488 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:38.177559 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:38.177867 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 04:20:38.177968 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 04:20:38.178556 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:38.168816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:36307 (local address 127.0.0.1:34108)
0810 04:20:38.168873 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:38.168886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:38.168929 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:38.169085 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:38.169086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:38.169278 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:38.169329 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.169331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.170291 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.170292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.171081 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:38.171083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:38.171514 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:38.171515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:38.171581 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:38.172061 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:38.172067 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:38.173691 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:38.175956 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.175957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.175958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.176198 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.177137 (+ 939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:38.177138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:38.177139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:38.177217 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:38.177566 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:38.177567 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:38.177654 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 04:20:38.178205 (+ 551us) client_negotiation.cc:240] Negotiation successful
0810 04:20:38.178414 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 04:20:38.179052 8009 catalog_manager.cc:1347] Generated new cluster ID: 257c2d2730624db19e772fc02d57e95c
I20260810 04:20:38.179098 8009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:38.180112 7881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552: Bootstrap starting.
I20260810 04:20:38.180791 7881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:38.181046 7881 log.cc:824] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:38.186216 7881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552: No bootstrap required, opened a new log
I20260810 04:20:38.188278 7881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
I20260810 04:20:38.188444 7881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1e93ea68d5400693181df052ced552, State: Initialized, Role: FOLLOWER
I20260810 04:20:38.188529 7881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } }
I20260810 04:20:38.188862 7881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:38.188921 7853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:38.189436 8018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.189620 8018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:38.190070 7853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [term 1 FOLLOWER]: Refusing update from remote peer d45c58beb16747f18e9a33dc1d59b157: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:38.190176 8009 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:38.190246 7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d45c58beb16747f18e9a33dc1d59b157" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0a1e93ea68d5400693181df052ced552"
I20260810 04:20:38.190474 8007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:38.190900 8009 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:38.190986 7854 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45c58beb16747f18e9a33dc1d59b157 in term 1.
I20260810 04:20:38.193447 8018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45c58beb16747f18e9a33dc1d59b157. Latest consensus state: current_term: 1 leader_uuid: "d45c58beb16747f18e9a33dc1d59b157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.193732 8018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:38.193809 8018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45c58beb16747f18e9a33dc1d59b157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.193897 8018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:38.195935 8029 mvcc.cc:205] Tried to move back new op lower bound from 7316830774022348800 to 7316830773831008256. Current Snapshot: MvccSnapshot[applied={T|T < 7316830773896155136}]
I20260810 04:20:38.198484 8033 catalog_manager.cc:1259] Loaded cluster ID: 257c2d2730624db19e772fc02d57e95c
I20260810 04:20:38.198570 8033 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552: loading cluster ID for follower catalog manager: success
I20260810 04:20:38.199716 8033 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552: acquiring CA information for follower catalog manager: success
W20260810 04:20:38.199893 8033 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260810 04:20:38.201339 8007 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45c58beb16747f18e9a33dc1d59b157" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d45c58beb16747f18e9a33dc1d59b157" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 45493 } } peers { permanent_uuid: "0a1e93ea68d5400693181df052ced552" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 35919 } } peers { permanent_uuid: "ee19159eda4447eca485131e9032ee93" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 36307 } } }
I20260810 04:20:38.202674 8007 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157 [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:38.202857 8009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d45c58beb16747f18e9a33dc1d59b157: Generated new TSK 0
I20260810 04:20:38.203034 8009 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 04:20:38.231757 7801 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.229746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.15.216.126:45493)
0810 04:20:38.229793 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.229795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.230859 (+ 1064us) server_negotiation.cc:407] Connection header received
0810 04:20:38.230969 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.230971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.230983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.231015 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.231679 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35658: BlockingRecv error: recv got EOF from 127.0.0.1:35658 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:20:38.233039 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.230125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.15.216.124:36307)
0810 04:20:38.230166 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.230168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.232410 (+ 2242us) server_negotiation.cc:407] Connection header received
0810 04:20:38.232614 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.232615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.232626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.232655 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.232978 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34124: BlockingRecv error: recv got EOF from 127.0.0.1:34124 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:20:38.234011 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.230485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.15.216.125:35919)
0810 04:20:38.230539 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.230541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.233403 (+ 2862us) server_negotiation.cc:407] Connection header received
0810 04:20:38.233502 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.233527 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.233539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.233569 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.233946 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56932: BlockingRecv error: recv got EOF from 127.0.0.1:56932 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260810 04:20:38.236547 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.235768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.15.216.125:35919)
0810 04:20:38.235804 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.235806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.236010 (+ 204us) server_negotiation.cc:407] Connection header received
0810 04:20:38.236120 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.236121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.236139 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.236168 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.236496 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56940: BlockingRecv error: recv got EOF from 127.0.0.1:56940 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:20:38.239696 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.238983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.15.216.125:35919)
0810 04:20:38.239017 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.239018 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.239211 (+ 193us) server_negotiation.cc:407] Connection header received
0810 04:20:38.239303 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.239303 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.239313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.239340 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.239649 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56950: BlockingRecv error: recv got EOF from 127.0.0.1:56950 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:20:38.243675 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.242968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.15.216.125:35919)
0810 04:20:38.243001 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.243003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.243194 (+ 191us) server_negotiation.cc:407] Connection header received
0810 04:20:38.243284 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.243285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.243294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.243322 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.243633 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56966: BlockingRecv error: recv got EOF from 127.0.0.1:56966 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 04:20:38.249722 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.248879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.15.216.125:35919)
0810 04:20:38.248914 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.248916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.249118 (+ 202us) server_negotiation.cc:407] Connection header received
0810 04:20:38.249210 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.249211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.249220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.249247 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.249572 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56982: BlockingRecv error: recv got EOF from 127.0.0.1:56982 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:20:38.259959 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.259073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.15.216.125:35919)
0810 04:20:38.259123 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.259125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.259368 (+ 243us) server_negotiation.cc:407] Connection header received
0810 04:20:38.259472 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.259473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.259486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.259516 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.259892 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56986: BlockingRecv error: recv got EOF from 127.0.0.1:56986 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 04:20:38.278650 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.277577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.15.216.125:35919)
0810 04:20:38.277704 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.277706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.277975 (+ 269us) server_negotiation.cc:407] Connection header received
0810 04:20:38.278084 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.278085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.278096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.278127 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.278589 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56992: BlockingRecv error: recv got EOF from 127.0.0.1:56992 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260810 04:20:38.313324 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.312355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.15.216.125:35919)
0810 04:20:38.312410 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.312412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.312691 (+ 279us) server_negotiation.cc:407] Connection header received
0810 04:20:38.312802 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.312803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.312815 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.312847 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.313261 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56998: BlockingRecv error: recv got EOF from 127.0.0.1:56998 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:20:38.379742 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.379003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.15.216.125:35919)
0810 04:20:38.379069 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.379071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.379200 (+ 129us) server_negotiation.cc:407] Connection header received
0810 04:20:38.379249 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.379249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.379260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.379325 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.379654 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57004: BlockingRecv error: recv got EOF from 127.0.0.1:57004 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 04:20:38.509743 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.509027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.15.216.125:35919)
0810 04:20:38.509101 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.509103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.509134 (+ 31us) server_negotiation.cc:407] Connection header received
0810 04:20:38.509207 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.509207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.509218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.509285 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.509681 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57006: BlockingRecv error: recv got EOF from 127.0.0.1:57006 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 04:20:38.767791 7885 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:38.767010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.15.216.125:35919)
0810 04:20:38.767098 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:38.767101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:38.767169 (+ 68us) server_negotiation.cc:407] Connection header received
0810 04:20:38.767235 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:38.767236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:38.767255 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:38.767318 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:38.767720 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57010: BlockingRecv error: recv got EOF from 127.0.0.1:57010 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:20:39.138468 7999 catalog_manager.cc:1259] Loaded cluster ID: 257c2d2730624db19e772fc02d57e95c
I20260810 04:20:39.138545 7999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: loading cluster ID for follower catalog manager: success
I20260810 04:20:39.139313 7999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: acquiring CA information for follower catalog manager: success
I20260810 04:20:39.139580 7999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee19159eda4447eca485131e9032ee93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:20:39.200572 8033 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a1e93ea68d5400693181df052ced552: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:20:39.284080 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:39.281149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.15.216.125:35919)
0810 04:20:39.281312 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:39.281314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:39.281426 (+ 112us) server_negotiation.cc:407] Connection header received
0810 04:20:39.281519 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:39.281520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:39.281666 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:39.281780 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:39.281951 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.282459 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.283240 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.283455 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.283529 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:39.283717 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:39.283828 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:39.283915 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:39.283956 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 04:20:39.283978 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260810 04:20:39.288421 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:39.286111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.15.216.124:36307)
0810 04:20:39.286249 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:39.286251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:39.286260 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:39.286297 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:39.286297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:39.286336 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:39.286412 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:39.286513 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.286985 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.287661 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.287871 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.287921 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:39.288057 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:39.288150 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:39.288266 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:39.288299 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 04:20:39.288322 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260810 04:20:39.294117 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:39.290696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.15.216.125:35919)
0810 04:20:39.290766 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:39.290768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:39.291011 (+ 243us) server_negotiation.cc:407] Connection header received
0810 04:20:39.291045 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:39.291046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:39.291059 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:39.291130 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:39.291241 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.291966 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.293295 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.293585 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.293683 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:39.293818 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:39.293853 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:39.293915 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:39.293990 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 04:20:39.294014 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:20:39.294389 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:39.290860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.15.216.124:36307)
0810 04:20:39.290897 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:39.290899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:39.291081 (+ 182us) server_negotiation.cc:407] Connection header received
0810 04:20:39.291200 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:39.291201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:39.291210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:39.291273 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:39.291505 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.292365 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.293418 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.293738 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.293822 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:39.294116 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:39.294149 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:39.294179 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:39.294297 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 04:20:39.294321 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:20:39.294824 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:39.290487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.15.216.126:45493)
0810 04:20:39.291674 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:39.291677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:39.291685 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:39.291732 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:39.291733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:39.291786 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:39.292049 (+ 263us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:39.292091 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.292791 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.293751 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:39.294093 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:39.294162 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:39.294357 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:39.294456 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:39.294687 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:39.294709 (+ 22us) server_negotiation.cc:299] Negotiation successful
0810 04:20:39.294737 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":57,"threads_started":1}
I20260810 04:20:39.296473 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7727
I20260810 04:20:39.306651 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7805
I20260810 04:20:39.313246 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 7889
2026-08-10T04:20:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2889 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 04:20:39 dist-test-slave-g350 krb5kdc[8127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 04:20:39 dist-test-slave-g350 krb5kdc[8127](info): set up 2 sockets
Aug 10 04:20:39 dist-test-slave-g350 krb5kdc[8127](info): commencing operation
krb5kdc: starting...
W20260810 04:20:40.395071 16225 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/test-user.keytab.
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T04:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T04:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.216.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
Entry for principal kudu/127.15.216.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab.
I20260810 04:20:40.524955 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:42487
--webserver_interface=127.15.216.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:42487,127.15.216.125:36959,127.15.216.124:44987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 04:20:40.616634 8141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:20:40.616918 8141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:40.616957 8141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:40.617007 8141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:40.618340 8141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:40.618417 8141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:40.618443 8141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:40.618481 8141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:40.618515 8141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:40.620190 8141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:42487,127.15.216.125:36959,127.15.216.124:44987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.126.keytab
--principal=kudu/127.15.216.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.126:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.216.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:40.620492 8141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:40.620745 8141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:40.625952 8149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:40.625982 8146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:40.625962 8147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:40.626197 8141 server_base.cc:1034] running on GCE node
I20260810 04:20:40.626547 8141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:40.626842 8141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:40.627991 8141 hybrid_clock.cc:648] HybridClock initialized: now 1786335640627970 us; error 45 us; skew 500 ppm
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:40.634595 8141 init.cc:377] Logged in from keytab as kudu/127.15.216.126@KRBTEST.COM (short username kudu)
I20260810 04:20:40.635064 8141 webserver.cc:460] Webserver started at http://127.15.216.126:35659/ using document root <none> and password file <none>
I20260810 04:20:40.635355 8141 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:40.635438 8141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:40.635591 8141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:40.636708 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data/instance:
uuid: "069c73917a8e4109999e31c28b3a438e"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "37d4c4bde038a43840496f8260708ed9"
server_key_iv: "3ee731a3bf07e630f530146471e1bfbe"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.637082 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/wal/instance:
uuid: "069c73917a8e4109999e31c28b3a438e"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "37d4c4bde038a43840496f8260708ed9"
server_key_iv: "3ee731a3bf07e630f530146471e1bfbe"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.638576 8141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:40.639386 8156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:40.639595 8141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 04:20:40.639712 8141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/wal
uuid: "069c73917a8e4109999e31c28b3a438e"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "37d4c4bde038a43840496f8260708ed9"
server_key_iv: "3ee731a3bf07e630f530146471e1bfbe"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.639789 8141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:40.653522 8141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:40.655900 8141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:40.656081 8141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:40.660482 8141 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.126:42487
I20260810 04:20:40.660512 8213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.126:42487 every 8 connection(s)
I20260810 04:20:40.660894 8141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/data/info.pb
I20260810 04:20:40.661854 8214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:40.663439 8214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:40.670867 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 8141
I20260810 04:20:40.670972 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-0/wal/instance
I20260810 04:20:40.671149 16225 external_mini_cluster.cc:1428] Setting key 1dfeee97ca128e126a6345a84a5aa4f3
I20260810 04:20:40.673466 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.664126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:42487 (local address 127.0.0.1:34952)
0810 04:20:40.664328 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.664370 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:40.664954 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:40.665243 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:40.665247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:40.665501 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:40.665721 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.665725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.666721 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.666722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.667383 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.667385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.667829 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.667830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.667888 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.668359 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:40.668367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:40.669876 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:40.671826 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.671834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.671836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.672121 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.672362 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.672363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.672363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.672476 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.672808 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:40.672815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:40.672890 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 04:20:40.672992 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 04:20:40.673313 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":149,"thread_start_us":94,"threads_started":1}
I20260810 04:20:40.673480 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.664196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.15.216.126:42487)
0810 04:20:40.664356 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:40.664365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:40.664437 (+ 72us) server_negotiation.cc:407] Connection header received
0810 04:20:40.665110 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:40.665112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:40.665129 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:40.665208 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:40.665824 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.666633 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.667460 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.667754 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.667836 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.669950 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:40.669957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:40.669958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:40.669964 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:40.671745 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.672208 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.672211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.672211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.672291 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.672542 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.672542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.672543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.672740 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:40.672802 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:40.673069 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 04:20:40.673237 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260810 04:20:40.674305 8214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:20:40.674679 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.674572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38314)
0810 04:20:40.674618 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.674660 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 04:20:40.674844 8160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
I20260810 04:20:40.675359 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.675223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38328)
0810 04:20:40.675295 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.675337 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 04:20:40.675514 8214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:36959: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.15.216.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab.
Entry for principal kudu/127.15.216.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab.
I20260810 04:20:40.700253 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:36959
--webserver_interface=127.15.216.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab
--principal=kudu/127.15.216.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:42487,127.15.216.125:36959,127.15.216.124:44987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:20:40.724759 8214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } attempt: 1
I20260810 04:20:40.725327 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.725147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38340)
0810 04:20:40.725223 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.725279 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 04:20:40.725816 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.725696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38352)
0810 04:20:40.725753 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.725788 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 04:20:40.725952 8214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:36959: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
W20260810 04:20:40.790237 8221 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:20:40.790454 8221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:40.790472 8221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:40.790493 8221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:40.791692 8221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:40.791738 8221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:40.791752 8221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:40.791764 8221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:40.791776 8221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:40.793264 8221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:42487,127.15.216.125:36959,127.15.216.124:44987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.125.keytab
--principal=kudu/127.15.216.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.125:36959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.216.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:40.793510 8221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:40.793771 8221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 04:20:40.794130 8214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } attempt: 2
I20260810 04:20:40.794633 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.794467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38362)
0810 04:20:40.794538 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.794591 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:20:40.795089 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.794972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38368)
0810 04:20:40.795027 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.795061 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 04:20:40.795200 8214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.125:36959: Network error: Client connection negotiation failed: client connection to 127.15.216.125:36959: connect: Connection refused (error 111)
W20260810 04:20:40.799016 8229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:40.799053 8221 server_base.cc:1034] running on GCE node
W20260810 04:20:40.799139 8227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:40.799016 8226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:40.799513 8221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:40.799818 8221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:40.800990 8221 hybrid_clock.cc:648] HybridClock initialized: now 1786335640800993 us; error 41 us; skew 500 ppm
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:40.803479 8221 init.cc:377] Logged in from keytab as kudu/127.15.216.125@KRBTEST.COM (short username kudu)
I20260810 04:20:40.803956 8221 webserver.cc:460] Webserver started at http://127.15.216.125:45195/ using document root <none> and password file <none>
I20260810 04:20:40.804199 8221 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:40.804270 8221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:40.804417 8221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:40.805534 8221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data/instance:
uuid: "fba1ccc0e23547ddb6324b26bee767e2"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "0044ace2523df29fa91891a88368047c"
server_key_iv: "f6ea68f044142fc78929cd04f076ab68"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.805975 8221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/wal/instance:
uuid: "fba1ccc0e23547ddb6324b26bee767e2"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "0044ace2523df29fa91891a88368047c"
server_key_iv: "f6ea68f044142fc78929cd04f076ab68"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.807312 8221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 04:20:40.808080 8236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:40.808255 8221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 04:20:40.808367 8221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/wal
uuid: "fba1ccc0e23547ddb6324b26bee767e2"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "0044ace2523df29fa91891a88368047c"
server_key_iv: "f6ea68f044142fc78929cd04f076ab68"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.808446 8221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:40.831297 8221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:40.833680 8221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:40.833870 8221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:40.838261 8221 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.125:36959
I20260810 04:20:40.838299 8293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.125:36959 every 8 connection(s)
I20260810 04:20:40.838821 8221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/data/info.pb
I20260810 04:20:40.839540 8294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:40.841059 8294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:40.847121 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 8221
I20260810 04:20:40.847224 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-1/wal/instance
I20260810 04:20:40.847450 16225 external_mini_cluster.cc:1428] Setting key 2a6e86c87817d8b58332bb82a9422e56
I20260810 04:20:40.850507 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.841725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.15.216.126:42487)
0810 04:20:40.841811 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:40.841813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:40.842035 (+ 222us) server_negotiation.cc:407] Connection header received
0810 04:20:40.842585 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:40.842603 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:40.842618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:40.842671 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:40.843323 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.843932 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.844851 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.845061 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.845117 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.847447 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:40.847448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:40.847448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:40.847454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:40.848934 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.849432 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.849433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.849434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.849505 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.849794 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.849794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.849795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.849958 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:40.850035 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:40.850273 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 04:20:40.850370 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 04:20:40.850528 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.841713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:42487 (local address 127.0.0.1:34966)
0810 04:20:40.841914 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.841951 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:40.842461 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:40.842730 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:40.842733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:40.843044 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:40.843241 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.843246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.844011 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.844012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.844773 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.844775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.845139 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.845140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.845239 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.845822 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:40.845830 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:40.847323 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:40.849022 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.849031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.849034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.849339 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.849576 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.849577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.849577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.849728 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.850030 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:40.850035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:40.850145 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 04:20:40.850225 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 04:20:40.850362 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260810 04:20:40.851197 8294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.216.124@KRBTEST.COM; defaulting to no policy
I20260810 04:20:40.858758 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.851482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38382)
0810 04:20:40.851531 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.851542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:40.851582 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:40.851830 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:40.851831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:40.852012 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:40.852085 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.852087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.852940 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.852941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.853840 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.853843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.854310 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.854311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.854397 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.854767 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:40.854772 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:40.856080 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:40.857890 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.857891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.857891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.858052 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.858241 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.858241 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.858242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.858293 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.858533 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:40.858534 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:40.858567 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 04:20:40.858619 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:20:40.858695 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 04:20:40.858832 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.851563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.15.216.125:36959)
0810 04:20:40.851691 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:40.851695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:40.851710 (+ 15us) server_negotiation.cc:407] Connection header received
0810 04:20:40.851725 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:40.851726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:40.851735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:40.851791 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:40.852175 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.852842 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.853933 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.854232 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.854287 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.856169 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:40.856179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:40.856180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:40.856187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:40.857817 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.858127 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.858129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.858129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.858182 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.858346 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.858347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.858347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.858477 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:40.858538 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:40.858660 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 04:20:40.858744 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260810 04:20:40.859517 8294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.216.124@KRBTEST.COM" created.
I20260810 04:20:40.859863 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.859755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38586)
0810 04:20:40.859786 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.859830 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 04:20:40.859984 8239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
I20260810 04:20:40.860401 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.860325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38596)
0810 04:20:40.860351 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.860377 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 04:20:40.860531 8294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:44987: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.216.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab.
Entry for principal kudu/127.15.216.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab.
I20260810 04:20:40.877482 16225 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrlBm0x/build/release/bin/kudu
/tmp/dist-test-taskrlBm0x/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:44987
--webserver_interface=127.15.216.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab
--principal=kudu/127.15.216.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.216.84:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.216.126:42487,127.15.216.125:36959,127.15.216.124:44987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 04:20:40.886346 8214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } attempt: 3
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:20:40.894166 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.886673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38386)
0810 04:20:40.886735 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.886749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:40.886801 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:40.886996 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:40.886997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:40.887147 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:40.887214 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.887216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.888217 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.888218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.888755 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:40.888756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.889266 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.889267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:40.889331 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.889927 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:40.889931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:40.891365 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:40.893130 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.893131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.893131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.893306 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.893510 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:40.893510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:40.893511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:40.893565 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:40.893904 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:40.893905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:40.893950 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 04:20:40.894009 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 04:20:40.894092 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 04:20:40.894238 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.886781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.15.216.125:36959)
0810 04:20:40.886836 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:40.886839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:40.886852 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:20:40.886884 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:40.886885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:40.886901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:40.886991 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:40.887295 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.888129 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.888838 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:40.889184 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:40.889253 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:40.891457 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:40.891459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:40.891459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:40.891467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:40.893036 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.893379 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.893380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.893380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.893443 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:40.893691 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:40.893692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:40.893692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:40.893836 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:40.893905 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:40.894054 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 04:20:40.894143 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 04:20:40.894731 8214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:20:40.895129 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.895007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38608)
0810 04:20:40.895057 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.895095 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 04:20:40.895522 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.895397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38622)
0810 04:20:40.895459 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.895490 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 04:20:40.895668 8214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:44987: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
I20260810 04:20:40.909760 8294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } attempt: 1
I20260810 04:20:40.910319 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.910188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38630)
0810 04:20:40.910248 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.910291 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:20:40.910707 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.910609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38632)
0810 04:20:40.910653 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.910687 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 04:20:40.910797 8294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:44987: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
I20260810 04:20:40.926828 8214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } attempt: 1
I20260810 04:20:40.927361 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.927193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38634)
0810 04:20:40.927266 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.927316 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 04:20:40.927745 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.927639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38648)
0810 04:20:40.927686 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.927715 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 04:20:40.927845 8214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:44987: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
W20260810 04:20:40.966549 8301 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 04:20:40.966769 8301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 04:20:40.966787 8301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 04:20:40.966807 8301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 04:20:40.967970 8301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 04:20:40.968016 8301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 04:20:40.968030 8301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 04:20:40.968043 8301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 04:20:40.968101 8301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 04:20:40.969699 8301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.216.84:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.216.126:42487,127.15.216.125:36959,127.15.216.124:44987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/krb5kdc/kudu_127.15.216.124.keytab
--principal=kudu/127.15.216.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.216.124:44987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.216.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 02:52:58 UTC on ccycloud.quasar-fnqwcq.root.comops.site
build id 4576984
I20260810 04:20:40.969951 8301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 04:20:40.970193 8301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 04:20:40.975468 8306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 04:20:40.975462 8307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:40.975574 8301 server_base.cc:1034] running on GCE node
W20260810 04:20:40.975458 8309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 04:20:40.975984 8301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 04:20:40.976325 8301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 04:20:40.977699 8301 hybrid_clock.cc:648] HybridClock initialized: now 1786335640977675 us; error 38 us; skew 500 ppm
I20260810 04:20:40.978943 8294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } attempt: 2
I20260810 04:20:40.979343 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.979213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38656)
0810 04:20:40.979276 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.979311 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 04:20:40.979774 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:40.979685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38662)
0810 04:20:40.979730 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:40.979756 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 04:20:40.979907 8294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.216.124:44987: Network error: Client connection negotiation failed: client connection to 127.15.216.124:44987: connect: Connection refused (error 111)
Aug 10 04:20:40 dist-test-slave-g350 krb5kdc[8127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 04:20:40.980616 8301 init.cc:377] Logged in from keytab as kudu/127.15.216.124@KRBTEST.COM (short username kudu)
I20260810 04:20:40.981106 8301 webserver.cc:460] Webserver started at http://127.15.216.124:46873/ using document root <none> and password file <none>
I20260810 04:20:40.981367 8301 fs_manager.cc:359] Metadata directory not provided
I20260810 04:20:40.981416 8301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 04:20:40.981570 8301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 04:20:40.982715 8301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data/instance:
uuid: "59276b80ae5945469b9590a1e4d35a15"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "06b1eb658e5b3c9e5889e69c4db1f487"
server_key_iv: "d017f8fe997f4ec5f5c0ac6703ae9444"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.983095 8301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/wal/instance:
uuid: "59276b80ae5945469b9590a1e4d35a15"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "06b1eb658e5b3c9e5889e69c4db1f487"
server_key_iv: "d017f8fe997f4ec5f5c0ac6703ae9444"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.984477 8301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 04:20:40.985191 8316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 04:20:40.985370 8301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 04:20:40.985476 8301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data,/tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/wal
uuid: "59276b80ae5945469b9590a1e4d35a15"
format_stamp: "Formatted at 2026-08-10 04:20:40 on dist-test-slave-g350"
server_key: "06b1eb658e5b3c9e5889e69c4db1f487"
server_key_iv: "d017f8fe997f4ec5f5c0ac6703ae9444"
server_key_version: "encryptionkey@0"
I20260810 04:20:40.985558 8301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 04:20:40.995572 8301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 04:20:40.997748 8301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 04:20:40.997918 8301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 04:20:41.002118 8301 rpc_server.cc:307] RPC server started. Bound to: 127.15.216.124:44987
I20260810 04:20:41.002247 8375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.216.124:44987 every 8 connection(s)
I20260810 04:20:41.002547 8301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/data/info.pb
I20260810 04:20:41.002993 16225 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrlBm0x/build/release/bin/kudu as pid 8301
I20260810 04:20:41.002970 8214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } attempt: 2
I20260810 04:20:41.003106 16225 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrlBm0x/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786335362448781-16225-0/minicluster-data/master-2/wal/instance
I20260810 04:20:41.003347 16225 external_mini_cluster.cc:1428] Setting key 2c9bc14fa47116b472a3ccb6679bdead
I20260810 04:20:41.003607 8376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 04:20:41.006240 8376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.126@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
I20260810 04:20:41.013904 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.003276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38668)
0810 04:20:41.003436 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:41.003449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:41.003502 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:41.004610 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:41.004611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:41.004798 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:41.004844 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.004847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.006034 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.006035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.006607 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.006609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.007422 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.007423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.007489 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.008003 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:41.008009 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:41.009541 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:41.012164 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.012165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.012166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.012426 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.012770 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.012771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.012771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.012838 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.013326 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:41.013327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:41.013377 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 04:20:41.013429 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 04:20:41.013789 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260810 04:20:41.014052 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.003672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.15.216.124:44987)
0810 04:20:41.003860 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.003862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.004042 (+ 180us) server_negotiation.cc:407] Connection header received
0810 04:20:41.004446 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.004447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.004480 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.004620 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.005516 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.006295 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.007343 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.007650 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.007711 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.010700 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.010701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.010702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.010709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.012627 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.012991 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.012992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.012992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.013051 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.013240 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.013241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.013242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.013355 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.013522 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.013640 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.013783 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":58,"threads_started":1}
I20260810 04:20:41.014236 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.003530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.15.216.124:44987)
0810 04:20:41.003843 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.003854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.003891 (+ 37us) server_negotiation.cc:407] Connection header received
0810 04:20:41.004425 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.004433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.004479 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.004620 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.004950 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.005933 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.006698 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.007357 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.007460 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.009708 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.009712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.009712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.009724 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.012087 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.012520 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.012524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.012524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.012658 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.012918 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.012919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.012920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.013227 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.013574 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.013651 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.014112 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":315,"server-negotiator.queue_time_us":247,"thread_start_us":90,"threads_started":1}
I20260810 04:20:41.016383 8214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e: Bootstrap starting.
I20260810 04:20:41.016758 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.007102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.15.216.126:42487)
0810 04:20:41.007143 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.007144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.007153 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:41.007207 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.007208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.007218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.007277 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.007823 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.008540 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.009703 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.010183 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.010228 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.013467 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.013468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.013469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.013474 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.015540 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.015894 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.015895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.015895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.015968 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.016161 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.016162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.016162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.016320 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.016395 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.016581 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.016662 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 04:20:41.016782 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.006890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.126:42487 (local address 127.0.0.1:34970)
0810 04:20:41.007052 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:41.007092 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:41.007150 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:41.007292 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:41.007295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:41.007657 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:41.007750 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.007753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.008634 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.008636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.009564 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.009567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.010318 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.010319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.010420 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.011102 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:41.011107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:41.013374 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:41.015630 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.015636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.015637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.015821 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.016042 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.016043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.016043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.016095 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.016392 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:41.016400 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:41.016461 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 04:20:41.016549 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 04:20:41.016662 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260810 04:20:41.017196 8214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:41.017259 8376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } has no permanent_uuid. Determining permanent_uuid...
I20260810 04:20:41.017527 8214 log.cc:824] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:41.018894 8214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e: No bootstrap required, opened a new log
I20260810 04:20:41.020556 8214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
I20260810 04:20:41.020694 8214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069c73917a8e4109999e31c28b3a438e, State: Initialized, Role: FOLLOWER
I20260810 04:20:41.020781 8214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
I20260810 04:20:41.021111 8214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:41.021085 8384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.021185 8384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.126@KRBTEST.COM
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
W20260810 04:20:41.023177 8396 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:20:41.023232 8396 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:20:41.025313 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.015138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.15.216.126:42487)
0810 04:20:41.016842 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.016844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.016852 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:41.016872 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.016873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.016883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.016940 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.017706 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.018198 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.019137 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.019340 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.019414 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.022002 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.022003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.022004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.022010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.023850 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.024254 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.024255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.024255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.024313 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.024536 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.024537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.024537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.024663 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.024780 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.025153 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.025232 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":94,"threads_started":1}
I20260810 04:20:41.027349 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.017759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.125:36959 (local address 127.0.0.1:38390)
0810 04:20:41.017814 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:41.017824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:41.017863 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:41.018572 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:41.018573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:41.018722 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:41.018774 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.018775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.019898 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.019899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.020734 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.020737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.021369 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.021370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.021442 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.021884 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:41.021889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:41.023366 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:41.026200 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.026201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.026202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.026429 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.026658 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.026659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.026659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.026726 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.027016 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:41.027017 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:41.027062 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 04:20:41.027149 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 04:20:41.027246 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 04:20:41.027367 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.018359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.15.216.125:36959)
0810 04:20:41.018440 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.018442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.018451 (+ 9us) server_negotiation.cc:407] Connection header received
0810 04:20:41.018476 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.018477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.018488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.018551 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.018867 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.019537 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.020862 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.021175 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.021250 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.023460 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.023462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.023462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.023469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.026104 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.026516 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.026517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.026518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.026577 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.026800 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.026801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.026801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.026937 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.027008 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.027189 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.027276 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 04:20:41.024910 8141 master_runner.cc:186] Error getting master registration for 127.15.216.125:36959: OK, instance_id { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" instance_seqno: 1786335640836888 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:20:41.027964 8141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:20:41.028482 8376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.124@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.216.125@KRBTEST.COM
I20260810 04:20:41.036218 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.028737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38680)
0810 04:20:41.028817 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:41.028826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:41.028872 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:41.028995 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:41.028996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:41.029142 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:41.029174 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.029175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.029938 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.029939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.030605 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.030607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.031095 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.031097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.031149 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.031557 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:41.031563 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:41.032734 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:41.035074 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.035075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.035076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.035237 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.035455 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.035456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.035456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.035530 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.035894 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:41.035895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:41.035941 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 04:20:41.036004 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 04:20:41.036095 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 04:20:41.036492 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.028779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.15.216.124:44987)
0810 04:20:41.028815 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.028816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.028864 (+ 48us) server_negotiation.cc:407] Connection header received
0810 04:20:41.028919 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.028920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.028929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.028985 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.029254 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.029849 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.030696 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.031014 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.031089 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.032819 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.032820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.032820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.032827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.035000 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.035311 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.035312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.035312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.035367 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.035635 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.035636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.035637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.035809 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.035871 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.036329 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.036414 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:20:41.037216 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.028617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.15.216.125:36959)
0810 04:20:41.028657 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.028659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.028667 (+ 8us) server_negotiation.cc:407] Connection header received
0810 04:20:41.028730 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.028731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.028740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.028789 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.029390 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.030097 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.030938 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.031207 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.031588 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.034143 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.034144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.034145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.034151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.035948 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.036347 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.036349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.036349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.036414 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.036611 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.036612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.036613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.036819 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.036993 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.037067 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.037140 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:20:41.038110 8376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: Bootstrap starting.
I20260810 04:20:41.038769 16225 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 04:20:41.038869 8376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:41.039150 8376 log.cc:824] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:41.039718 8186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "069c73917a8e4109999e31c28b3a438e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34986
I20260810 04:20:41.039872 8186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 04:20:41.039934 8186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:41.040035 8376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: No bootstrap required, opened a new log
I20260810 04:20:41.040912 8186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
I20260810 04:20:41.041175 8186 leader_election.cc:288] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [CANDIDATE]: Term 1 election: Requested vote from peers fba1ccc0e23547ddb6324b26bee767e2 (127.15.216.125:36959), 59276b80ae5945469b9590a1e4d35a15 (127.15.216.124:44987)
I20260810 04:20:41.042270 8376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
I20260810 04:20:41.042448 8376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59276b80ae5945469b9590a1e4d35a15, State: Initialized, Role: FOLLOWER
I20260810 04:20:41.042541 8376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
I20260810 04:20:41.042869 8376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:41.042902 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "069c73917a8e4109999e31c28b3a438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59276b80ae5945469b9590a1e4d35a15"
I20260810 04:20:41.043020 8346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:41.043920 8346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069c73917a8e4109999e31c28b3a438e in term 1.
I20260810 04:20:41.044926 8161 leader_election.cc:302] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069c73917a8e4109999e31c28b3a438e, 59276b80ae5945469b9590a1e4d35a15; no voters:
I20260810 04:20:41.045063 8384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 04:20:41.045220 8384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [term 1 LEADER]: Becoming Leader. State: Replica: 069c73917a8e4109999e31c28b3a438e, State: Running, Role: LEADER
I20260810 04:20:41.045321 8384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
W20260810 04:20:41.045557 8418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:20:41.045735 8418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:20:41.046036 8384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069c73917a8e4109999e31c28b3a438e. Latest consensus state: current_term: 1 leader_uuid: "069c73917a8e4109999e31c28b3a438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.046113 8384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:41.046341 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.046634 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:41.048517 8422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 04:20:41.048709 8422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 04:20:41.047039 8301 master_runner.cc:186] Error getting master registration for 127.15.216.125:36959: OK, instance_id { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" instance_seqno: 1786335640836888 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 04:20:41.049490 8301 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 04:20:41.053946 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.042364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.15.216.126:42487)
0810 04:20:41.042405 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.042407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.042924 (+ 517us) server_negotiation.cc:407] Connection header received
0810 04:20:41.043043 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.043044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.043059 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.043093 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.043981 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.044736 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.047762 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.048075 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.048120 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.050931 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.050932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.050933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.050940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.052597 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.053021 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.053022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.053022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.053082 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.053297 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.053298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.053298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.053425 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.053496 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.053756 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.053838 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 04:20:41.056977 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.042676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.15.216.124:44987)
0810 04:20:41.042707 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.042708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.044203 (+ 1495us) server_negotiation.cc:407] Connection header received
0810 04:20:41.044390 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.044391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.044402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.044436 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.045731 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.046454 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.048063 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.048389 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.048778 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.052568 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.052570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.052570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.052578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.054379 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.055424 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.055425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.055426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.055496 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.056002 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.056003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.056003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.056149 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.056231 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.056801 (+ 570us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.056886 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 04:20:41.058611 8346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [term 1 FOLLOWER]: Refusing update from remote peer 069c73917a8e4109999e31c28b3a438e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:41.058972 8419 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 04:20:41.060300 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069c73917a8e4109999e31c28b3a438e. Latest consensus state: current_term: 1 leader_uuid: "069c73917a8e4109999e31c28b3a438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.060395 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:41.062014 8384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069c73917a8e4109999e31c28b3a438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.062098 8384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:41.062522 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069c73917a8e4109999e31c28b3a438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.062604 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:41.063659 8425 mvcc.cc:205] Tried to move back new op lower bound from 7316830785772998656 to 7316830785722269696. Current Snapshot: MvccSnapshot[applied={T|T < 7316830785772998656}]
I20260810 04:20:41.065011 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.046191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.15.216.125:36959)
0810 04:20:41.046236 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.046238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.054301 (+ 8063us) server_negotiation.cc:407] Connection header received
0810 04:20:41.054423 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.054424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.054437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.054467 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.056745 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.057513 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.058930 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.059230 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.059276 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.061483 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.061484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.061485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.061503 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.063287 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.063881 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.063882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.063883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.063953 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.064250 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.064251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.064251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.064401 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.064487 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.064829 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.064916 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 04:20:41.070060 8422 catalog_manager.cc:1347] Generated new cluster ID: 57794c4d3da24d95a6c41ab643536fce
I20260810 04:20:41.070115 8422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 04:20:41.071045 8294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } attempt: 3
Aug 10 04:20:41 dist-test-slave-g350 krb5kdc[8127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786335640, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.216.125@KRBTEST.COM for kudu/127.15.216.124@KRBTEST.COM
I20260810 04:20:41.078310 8422 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 04:20:41.079110 8422 catalog_manager.cc:1504] Loading token signing keys...
I20260810 04:20:41.079702 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.071245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.216.124:44987 (local address 127.0.0.1:38690)
0810 04:20:41.071323 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 04:20:41.071345 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 04:20:41.071380 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 04:20:41.071604 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 04:20:41.071605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 04:20:41.071762 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 04:20:41.071801 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.071802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.072787 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.072788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.073613 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 04:20:41.073616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.074152 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.074153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 04:20:41.074223 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.074745 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 04:20:41.074750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 04:20:41.076307 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 04:20:41.078444 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.078445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.078446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.078676 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.078941 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 04:20:41.078942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 04:20:41.078942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 04:20:41.079015 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 04:20:41.079379 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 04:20:41.079380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 04:20:41.079440 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 04:20:41.079504 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 04:20:41.079603 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 04:20:41.080180 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:41.071449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.15.216.124:44987)
0810 04:20:41.071483 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.071485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.071491 (+ 6us) server_negotiation.cc:407] Connection header received
0810 04:20:41.071511 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.071512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.071522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.071565 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.071879 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.072616 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.073722 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:41.074024 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:41.074082 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:41.076429 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 04:20:41.076430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 04:20:41.076431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 04:20:41.076437 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 04:20:41.078285 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.078779 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.078780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.078780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.078854 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 04:20:41.079094 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 04:20:41.079095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 04:20:41.079096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 04:20:41.079233 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 04:20:41.079293 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:41.080015 (+ 722us) server_negotiation.cc:299] Negotiation successful
0810 04:20:41.080092 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 04:20:41.082207 8294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: Bootstrap starting.
I20260810 04:20:41.082955 8294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: Neither blocks nor log segments found. Creating new log.
I20260810 04:20:41.083262 8294 log.cc:824] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: Log is configured to *not* fsync() on all Append() calls
I20260810 04:20:41.084086 8422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e: Generated new TSK 0
I20260810 04:20:41.084259 8422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 04:20:41.093811 8294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: No bootstrap required, opened a new log
I20260810 04:20:41.095964 8294 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
I20260810 04:20:41.096097 8294 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba1ccc0e23547ddb6324b26bee767e2, State: Initialized, Role: FOLLOWER
I20260810 04:20:41.096171 8294 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } }
I20260810 04:20:41.096501 8294 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 04:20:41.096818 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "069c73917a8e4109999e31c28b3a438e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fba1ccc0e23547ddb6324b26bee767e2"
I20260810 04:20:41.096941 8266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 04:20:41.097873 8266 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 069c73917a8e4109999e31c28b3a438e in term 1.
I20260810 04:20:41.098310 8265 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [term 1 FOLLOWER]: Refusing update from remote peer 069c73917a8e4109999e31c28b3a438e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 04:20:41.098706 8428 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 04:20:41.099411 8445 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 04:20:41.099455 8445 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 04:20:41.099572 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.099637 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:41.102041 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069c73917a8e4109999e31c28b3a438e. Latest consensus state: current_term: 1 leader_uuid: "069c73917a8e4109999e31c28b3a438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.102203 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:41.102280 8432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "069c73917a8e4109999e31c28b3a438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.102391 8432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 04:20:41.106206 8447 mvcc.cc:205] Tried to move back new op lower bound from 7316830785875935232 to 7316830785722269696. Current Snapshot: MvccSnapshot[applied={T|T < 7316830785772998656}]
W20260810 04:20:41.123369 8379 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.121360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.15.216.124:44987)
0810 04:20:41.121437 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.121439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.122626 (+ 1187us) server_negotiation.cc:407] Connection header received
0810 04:20:41.122747 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.122748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.122761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.122792 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.123291 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38696: BlockingRecv error: recv got EOF from 127.0.0.1:38696 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 04:20:41.124146 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.121889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.15.216.125:36959)
0810 04:20:41.121932 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.121934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.123620 (+ 1686us) server_negotiation.cc:407] Connection header received
0810 04:20:41.123717 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.123718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.123729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.123757 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.124073 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38416: BlockingRecv error: recv got EOF from 127.0.0.1:38416 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 04:20:41.124867 8217 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.122264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.15.216.126:42487)
0810 04:20:41.122306 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.122308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.124362 (+ 2054us) server_negotiation.cc:407] Connection header received
0810 04:20:41.124465 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.124466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.124478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.124508 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.124801 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35010: BlockingRecv error: recv got EOF from 127.0.0.1:35010 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260810 04:20:41.127337 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.126180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.15.216.125:36959)
0810 04:20:41.126213 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.126215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.126819 (+ 604us) server_negotiation.cc:407] Connection header received
0810 04:20:41.126939 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.126940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.126958 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.126987 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.127292 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38424: BlockingRecv error: recv got EOF from 127.0.0.1:38424 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 04:20:41.130295 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.129530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.15.216.125:36959)
0810 04:20:41.129561 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.129562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.129808 (+ 246us) server_negotiation.cc:407] Connection header received
0810 04:20:41.129916 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.129917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.129927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.129955 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.130253 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38428: BlockingRecv error: recv got EOF from 127.0.0.1:38428 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 04:20:41.134146 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.133375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.15.216.125:36959)
0810 04:20:41.133409 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.133410 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.133671 (+ 261us) server_negotiation.cc:407] Connection header received
0810 04:20:41.133763 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.133764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.133773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.133801 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.134107 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38438: BlockingRecv error: recv got EOF from 127.0.0.1:38438 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 04:20:41.139979 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.139231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.15.216.125:36959)
0810 04:20:41.139265 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.139267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.139493 (+ 226us) server_negotiation.cc:407] Connection header received
0810 04:20:41.139584 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.139585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.139595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.139622 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.139940 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38446: BlockingRecv error: recv got EOF from 127.0.0.1:38446 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 04:20:41.150079 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.149156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.15.216.125:36959)
0810 04:20:41.149212 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.149213 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.149499 (+ 286us) server_negotiation.cc:407] Connection header received
0810 04:20:41.149624 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.149626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.149637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.149668 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.150031 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38458: BlockingRecv error: recv got EOF from 127.0.0.1:38458 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260810 04:20:41.168342 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.167411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.15.216.125:36959)
0810 04:20:41.167454 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.167456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.167726 (+ 270us) server_negotiation.cc:407] Connection header received
0810 04:20:41.167834 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.167836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.167848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.167891 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.168285 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38474: BlockingRecv error: recv got EOF from 127.0.0.1:38474 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:20:41.202744 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.201806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.15.216.125:36959)
0810 04:20:41.201852 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.201854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.202144 (+ 290us) server_negotiation.cc:407] Connection header received
0810 04:20:41.202248 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.202249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.202262 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.202291 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.202680 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38480: BlockingRecv error: recv got EOF from 127.0.0.1:38480 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 04:20:41.268965 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.268250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.15.216.125:36959)
0810 04:20:41.268309 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.268311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.268415 (+ 104us) server_negotiation.cc:407] Connection header received
0810 04:20:41.268487 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.268488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.268500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.268561 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.268900 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38484: BlockingRecv error: recv got EOF from 127.0.0.1:38484 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 04:20:41.398875 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.398183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.15.216.125:36959)
0810 04:20:41.398267 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.398269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.398374 (+ 105us) server_negotiation.cc:407] Connection header received
0810 04:20:41.398436 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.398437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.398448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.398493 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.398803 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38500: BlockingRecv error: recv got EOF from 127.0.0.1:38500 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 04:20:41.656904 8298 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 04:20:41.656242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.15.216.125:36959)
0810 04:20:41.656317 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:41.656319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:41.656330 (+ 11us) server_negotiation.cc:407] Connection header received
0810 04:20:41.656392 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:41.656393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:41.656404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:41.656464 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 04:20:41.656825 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38510: BlockingRecv error: recv got EOF from 127.0.0.1:38510 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 04:20:41.706079 8428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "069c73917a8e4109999e31c28b3a438e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069c73917a8e4109999e31c28b3a438e" member_type: VOTER last_known_addr { host: "127.15.216.126" port: 42487 } } peers { permanent_uuid: "fba1ccc0e23547ddb6324b26bee767e2" member_type: VOTER last_known_addr { host: "127.15.216.125" port: 36959 } } peers { permanent_uuid: "59276b80ae5945469b9590a1e4d35a15" member_type: VOTER last_known_addr { host: "127.15.216.124" port: 44987 } } }
I20260810 04:20:41.706244 8428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069c73917a8e4109999e31c28b3a438e [sys.catalog]: This master's current role is: LEADER
I20260810 04:20:42.046218 8418 catalog_manager.cc:1259] Loaded cluster ID: 57794c4d3da24d95a6c41ab643536fce
I20260810 04:20:42.046317 8418 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: loading cluster ID for follower catalog manager: success
I20260810 04:20:42.047148 8418 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: acquiring CA information for follower catalog manager: success
I20260810 04:20:42.047489 8418 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59276b80ae5945469b9590a1e4d35a15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:20:42.099923 8445 catalog_manager.cc:1259] Loaded cluster ID: 57794c4d3da24d95a6c41ab643536fce
I20260810 04:20:42.100015 8445 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: loading cluster ID for follower catalog manager: success
I20260810 04:20:42.100867 8445 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: acquiring CA information for follower catalog manager: success
I20260810 04:20:42.101202 8445 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fba1ccc0e23547ddb6324b26bee767e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 04:20:42.172744 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:42.170194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.15.216.125:36959)
0810 04:20:42.170338 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:42.170340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:42.170353 (+ 13us) server_negotiation.cc:407] Connection header received
0810 04:20:42.170423 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:42.170424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:42.170476 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:42.170560 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:42.170701 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.171215 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.171957 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.172154 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.172211 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:42.172410 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:42.172535 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:42.172621 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:42.172650 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 04:20:42.172670 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260810 04:20:42.177338 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:42.174784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.15.216.124:44987)
0810 04:20:42.174909 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:42.174911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:42.174921 (+ 10us) server_negotiation.cc:407] Connection header received
0810 04:20:42.175003 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:42.175004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:42.175059 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:42.175144 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:42.175271 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.175762 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.176500 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.176725 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.176802 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:42.176976 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:42.177093 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:42.177182 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:42.177222 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 04:20:42.177243 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260810 04:20:42.182080 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:42.179155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.15.216.125:36959)
0810 04:20:42.179221 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:42.179223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:42.179462 (+ 239us) server_negotiation.cc:407] Connection header received
0810 04:20:42.179528 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:42.179530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:42.179542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:42.179602 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:42.179734 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.180451 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.181334 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.181547 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.181648 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:42.181808 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:42.181840 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:42.181866 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:42.181980 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 04:20:42.182002 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 04:20:42.182638 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:42.179210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.15.216.124:44987)
0810 04:20:42.179249 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:42.179251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:42.179599 (+ 348us) server_negotiation.cc:407] Connection header received
0810 04:20:42.179691 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:42.179692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:42.179704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:42.179786 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:42.179958 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.180475 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.181764 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.181971 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.182005 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:42.182234 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:42.182287 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:42.182320 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:42.182530 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 04:20:42.182555 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 04:20:42.183038 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 04:20:42.179066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.15.216.126:42487)
0810 04:20:42.179184 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 04:20:42.179186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 04:20:42.179313 (+ 127us) server_negotiation.cc:407] Connection header received
0810 04:20:42.179392 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 04:20:42.179393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 04:20:42.179429 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 04:20:42.179528 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 04:20:42.179655 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.180450 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.181760 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 04:20:42.182068 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 04:20:42.182137 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 04:20:42.182525 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 04:20:42.182767 (+ 242us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 04:20:42.182826 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 04:20:42.182924 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 04:20:42.182949 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260810 04:20:42.184756 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 8141
I20260810 04:20:42.195050 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 8221
I20260810 04:20:42.203258 16225 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrlBm0x/build/release/bin/kudu with pid 8301
2026-08-10T04:20:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2891 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5780 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279763 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST